New models connect internet knowledge to robot movement

Researchers tracked the development of vision-language-action models from 2022 to 2026.

Vision-language-action (VLA) models help robots use knowledge from the web to control physical movements. These models adapt existing vision and language systems to produce robot actions. This allows web-based knowledge to carry over to physical control.

In the past, robot learning required a separate policy for every specific robot and task. These older datasets were much smaller than those used for web-based models. VLA models bridge this gap by connecting the two types of learning.

The field has changed rapidly since the first VLA, called RT-2, appeared. Researchers have moved from early models to proprietary systems that clean homes. They have also created open models that work on a single consumer GPU.

Three main trends have emerged in this research. First, training data has expanded significantly. It grew from 10,000 hours of teleoperated robot data in π0\pi_{0} to 23,000 community-collected episodes in SmolVLA. It also includes 20,000 hours of human video in GR00T N1.7.

Second, model sizes are splitting into two groups. Most recent models use 3 to 7 billion parameters. However, a new group uses under one billion parameters to run on consumer GPUs. Third, generalist VLAs like UniVLA and Qwen-VLA can now handle both navigation and manipulation.

A VLA model uses several design elements. Inputs and memory describe what the model observes. Architecture refers to how the pipeline is built. Action output is how the model produces motor commands. Intermediate predictions include reasoning steps or future observations.

Training involves pretraining data and post-training. Pretraining data is a mix of robot, web, human, and simulated data. Post-training is how the model adapts to a specific robot. Deployment requires real-time execution to keep pace with the robot.

Some VLA models show near-perfect scores on benchmarks. However, these models can fail if a camera or an object moves slightly. This finding opens up further research questions.

Key facts

  • Training data grew from 10,000 hours of robot data to 20,000 hours of human video.
  • Newer models can run on consumer GPUs using under one billion parameters.
  • Generalist models like UniVLA can perform both navigation and manipulation tasks.

All sections

Edition