New defense stops hidden malicious behaviors in digital agents

Researchers developed a method to prevent dormant poisoning attacks from activating during fine-tuning.

Attackers can hide malicious behaviors that stay secret until benign fine-tuning occurs. These dormant poisoning attacks can extend to unseen actions or multiple harmful behaviors. This risk applies to language models, language agents, and multimodal agents.

The researchers created a Safety Buffer to protect against these risks. A Safety Buffer pairs harmful inputs with responses from the released model and benign versions. This method uses the fact that attackers must keep behavior safe at release to hide poisoning.

They also introduced LookAhead Defense to improve existing methods. LookAhead Defense previews benign fine-tuning updates using the Safety Buffer. It penalizes updates predicted to weaken safe behavior or increase refusals on benign versions.

LookAhead Defense reduces harmful behavior in agents from up to 93.9%93.9\% to at most 14.2%14.2\%. It reduces harmful responses in language models from up to 92.5%92.5\% to at most 2.0%2.0\% compared to vanilla fine-tuning. The method maintains competitive utility and low over-refusal.

Key facts

  • Harmful behaviors can be activated by benign fine-tuning.
  • LookAhead Defense reduces harmful agent behavior from up to 93.9%93.9\% to at most 14.2%14.2\%.
  • The defense works across language models, language agents, and multimodal agents.

All sections

Edition