New method creates realistic radar data from moving scenes
4D Point Splatting fits moving scenes to real captures to help train radar models.
4D Point Splatting (4DPS) is a physics-based differentiable renderer. It fits an explicit moving scene to a real capture.
Millimeter-wave radar resolves micrometer displacements of a moving body as micro-Doppler. This enables sensing of gait, gesture, activity, pose and vital-signs.
The method represents a scene as oriented points. These points carry material, normal and velocity. The points splat into range and Doppler bins.
The output is the raw complex return a real radar records. This allows signal processing to obtain representations for training any state of the art (SOTA) downstream model.
On the RT-Pose dataset, a pose model trained on these renders reaches cm MPJPE on real data. This compares to cm for real training and cm for the RF-Genesis baseline.
Custom CUDA kernels fit a sequence in minutes on HuPR and on RT-Pose using one H100. This makes dataset-scale inverse rendering of mmWave radar possible for the first time.
Key facts
- 4DPS achieved median correlations of and on micro-Doppler.
- The method uses a synchronized camera to initialize the room and body.
- This work can enable synthetic data generation for human-centered tasks and scene editing for data augmentation.