A new model helps robots connect language, video, and actions

The SWAM model uses large amounts of video and demonstration data to improve how robots perform tasks.

Researchers developed SWAM, which stands for Scaling World Action Models with Cross-Modal Alignment. This model predicts video and actions based on language.

The team used 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. They studied how language-video and language-video-action supervision affect performance.

World action models jointly predict video and actions conditioned on language. Language-video alignment is the connection between language and video acquired through video pretraining.

Mixing egocentric videos and UMI demonstrations improves control generalization under a fixed training budget. The model was post-trained on RoboTwin Clean demonstrations only.

The model achieved success rates of 88.10%88.10\% and 61.56%61.56\% under Easy and Hard conditions. It reached a 40.00%40.00\% success rate across three real-world tasks.

The model shows better generalization under unseen conditions compared to the strongest baseline of 23.33%23.33\%. The researchers released the curated data, training code, and model weights.

Key facts

  • The model uses 47,000 hours of various video and demonstration data.
  • SWAM achieved a 40.00%40.00\% success rate in three real-world tasks.
  • The model outperforms the strongest baseline of 23.33%23.33\% in real-world tasks.

All sections

Edition