A new framework for humanoid robots to join social activities

The SocialCoAct framework helps robots coordinate speech and actions during human interactions.

Researchers created SocialCoAct to help humanoid robots participate in social interactions. This framework enables robots to use vision and language for social co-action.

The SocialCoAct-Agent system turns observations and dialogue history into verbal responses and atomic action sequences. It aligns each action with its corresponding utterance span. The robot then executes these behaviors in MuJoCo and on a physical humanoid robot.

Egocentric social co-action is an end-to-end agent task. Egocentric observations are views from the perspective of the agent. Atomic action sequences are sets of individual actions.

To test these abilities, researchers built SocialCoAct-Bench using recordings of natural multi-party interactions. This benchmark includes 423 human-verified multiple-choice questions. These questions cover Long Episode Understanding (LEU), Speaker Intention Understanding (SIU), and Coordinated Action Planning (CAP).

The highest overall accuracy across different models was 77.54%. Accuracy on the hard subset reached only 46.51%. Using extended reasoning improved Qwen3-VL-2B and Qwen3-VL-4B by 19.62 and 17.02 percentage points, but it reduced inference throughput by 53.6×\times and 20.2×\times.

Key facts

  • SocialCoAct-Bench contains 423 human-verified multiple-choice questions.
  • The highest overall accuracy for tested models is 77.54%.
  • Extended reasoning reduces inference throughput by 53.6×\times for Qwen3-VL-2B.

All sections

Edition