New method uses human videos to help robots learn tasks

A system called SALA aligns human movements with robot actions using language descriptions.

Researchers created SALA to help robots learn from videos of humans. This method helps robots understand how to perform tasks by watching people.

Collecting robot interaction data is costly. Egocentric human videos can provide extra experience. However, human and robot movements look different. This makes it hard to transfer knowledge between them.

Latent action models (LAMs) use visual changes to represent actions. They are more efficient than world-action models (WAMs). WAMs predict future video frames and actions. LAMs avoid predicting pixels during learning and execution.

Most LAMs learn by reconstructing videos. This can cause them to group things by how they look instead of the action. This makes it hard to match human actions to robot actions.

SALA uses language to fix this. It uses text descriptions of actions like "grasp the cup". Even if human and robot videos are unpaired, these descriptions show semantic similarities. This guides human latent actions toward related robot latents.

The system uses a latent action tokenizer. This tool labels videos using only their frames. It provides a signal for training the robot policy. This allows scaling along two axes.

First, pre-training on general human videos reduces robot action prediction error. Second, post-training on task-specific human videos improves success rates. In tests, post-training raised the real-robot success rate from 53.75% to 72.5%.

The researchers tested SALA on a 63-DoF real robot. This robot includes two 22-DoF dexterous hands. They also tested it in a simulation called RoboTwin 2.0.

Key facts

  • SALA uses language descriptions to align human and robot actions.
  • Post-training on task-specific human videos increased real-robot success from 53.75% to 72.5%.
  • The method allows robots to learn from large amounts of general human video.

All sections

Edition