New vision models help robots predict tennis movements

Researchers tested how vision models use egocentric video to track players and balls in tennis.

Researchers tested how vision models use egocentric video to track players and balls in tennis. These models can predict ball trajectories and player movements.

The team collected an annotated tennis dataset. This dataset includes egocentric video synchronized with five allocentric videos. It also contains estimated global positions, labeled contact times, and annotated stroke types.

Egocentric video is a natural view for robots. The study evaluated seven prediction tasks. These tasks include 2D ego player and 3D ball localization. They also include predicting future player and ball trajectories.

The best models achieved a mean current-frame localization error of 0.140.14 m for the ego player and 0.540.54 m for the ball. Future forecasting errors were 0.340.34 m for the player and 0.910.91 m for the ball over 1.11.1 s. The mean contact-time prediction error was 0.140.14 s.

The results suggest that inferring metric depth remains a challenge. Longer image histories do not improve ball forecasting. This indicates that representations exploit scene context rather than relying solely on inter-frame motion.

A conditional diffusion model was proposed. This model generates whole-body kinematic tennis motion from vision encoder predictions at real-time rates. It adapts movements as predictions are refined.

Key facts

  • The best models had a mean contact-time prediction error of 0.140.14 s.
  • Future forecasting errors were 0.340.34 m for the player and 0.910.91 m for the ball over 1.11.1 s.
  • Auxiliary depth information substantially improves 3D ball localization.

All sections

Edition