SWAM: Scaling World Action Models with Cross-Modal Alignment
Overview
This research paper introduces SWAM, a world action model that jointly predicts video and actions conditioned on language. The authors address the challenge of maintaining language-video alignment while introducing action learning by curating approximately 47,000 hours of diverse data including general videos, egocentric videos, UMI demonstrations, and robot demonstrations. They systematically study how different supervision strategies and model architectures affect performance. Under a fixed training budget, mixing egocentric videos and UMI demonstrations improves control generalization. The resulting model achieves 88.10% success on Easy (Clean-to-Clean) and 61.56% on Hard (Clean-to-Random) conditions in simulation, and 40.00% success across three real-world tasks compared to 23.33% for the strongest baseline. The authors release the curated data, training code, and model weights.
Generated from attributed reports · Updated 3 hours ago
Event evidence and corrections
0 attributed source owners. Ownership does not establish independent confirmation. Quantities are reported separately and are never added together.
Reported quantity · Reported duration: 47000 hours · Basis not reported
“design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In partic”
Exact source · revision 1Source owner not reported
Reported quantity · Reported success rate: 88.1 percent · reported trials · Differing source assertions
“control generalization. Model architecture affects how effectively these experiences are used, while scaling pretraining under the selected configuration brings further gains. The resulting recipe, post-trained on RoboTwin Clean demonstrations only, achieves success rates of 88.10% and 61.56% under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. Our model achieves 40.00% SR across three rea”
Exact source · revision 1Source owner not reported
Reported quantity · Reported success rate: 61.56 percent · reported trials · Differing source assertions
“control generalization. Model architecture affects how effectively these experiences are used, while scaling pretraining under the selected configuration brings further gains. The resulting recipe, post-trained on RoboTwin Clean demonstrations only, achieves success rates of 88.10% and 61.56% under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. Our model achieves 40.00% SR across three rea”
Exact source · revision 1Source owner not reported
Artifact availability · code: reported pending
“l-world tasks, compared with 23.33% for the strongest baseline, indicating better generalization under unseen conditions. We release the curated data, training code, and model weights.”
Exact source · revision 1Source owner not reported
Artifact availability · weights: reported pending
“l-world tasks, compared with 23.33% for the strongest baseline, indicating better generalization under unseen conditions. We release the curated data, training code, and model weights.”
Exact source · revision 1Source owner not reported
Dataset · modalities: video · Differing source assertions
“design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In partic”
Exact source · revision 1Source owner not reported
Dataset · modalities: action · Differing source assertions
“design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In partic”
Exact source · revision 1Source owner not reported
Dataset · embodiments: general videos · Differing source assertions
“design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In partic”
Exact source · revision 1Source owner not reported
Dataset · embodiments: egocentric videos · Differing source assertions
“design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In partic”
Exact source · revision 1Source owner not reported
Dataset · embodiments: UMI demonstrations · Differing source assertions
“design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In partic”
Exact source · revision 1Source owner not reported
Dataset · embodiments: robot demonstrations · Differing source assertions
“design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In partic”
Exact source · revision 1Source owner not reported
Report timeline
Follow attributed reports and material updates.
- arXiv Robotics — research abstractsSignalSWAM: Scaling World Action Models with Cross-Modal Alignment
This research paper introduces SWAM, a world action model that jointly predicts video and actions conditioned on language. The authors address the challenge of maintaining language-video alignment while introducing action learning by curating approximately 47,000 hours of diverse data including general videos, egocentric videos, UMI demonstrations, and robot demonstrations. They systematically study how different supervision strategies and model architectures affect performance. Under a fixed training budget, mixing egocentric videos and UMI demonstrations improves control generalization. The resulting model achieves 88.10% success on Easy (Clean-to-Clean) and 61.56% on Hard (Clean-to-Random) conditions in simulation, and 40.00% success across three real-world tasks compared to 23.33% for the strongest baseline. The authors release the curated data, training code, and model weights.
Event coverage history
There is not enough continuous observation data to show a trend.
Timezone · UTC
Article dates follow your selected timezone. Briefing editions use Hong Kong time (UTC+8).