New method creates realistic radar data from moving scenes

4D Point Splatting fits moving scenes to real captures to help train radar models.

4D Point Splatting (4DPS) is a physics-based differentiable renderer. It fits an explicit moving scene to a real capture.

Millimeter-wave radar resolves micrometer displacements of a moving body as micro-Doppler. This enables sensing of gait, gesture, activity, pose and vital-signs.

The method represents a scene as oriented points. These points carry material, normal and velocity. The points splat into range and Doppler bins.

The output is the raw complex return a real radar records. This allows signal processing to obtain representations for training any state of the art (SOTA) downstream model.

On the RT-Pose dataset, a pose model trained on these renders reaches 21.921.9 cm MPJPE on real data. This compares to 16.716.7 cm for real training and 127.6127.6 cm for the RF-Genesis baseline.

Custom CUDA kernels fit a sequence in 5.75.7 minutes on HuPR and 12.912.9 on RT-Pose using one H100. This makes dataset-scale inverse rendering of mmWave radar possible for the first time.

Key facts

  • 4DPS achieved median correlations of 0.9880.988 and 0.9660.966 on micro-Doppler.
  • The method uses a synchronized camera to initialize the room and body.
  • This work can enable synthetic data generation for human-centered tasks and scene editing for data augmentation.

All sections

Edition