New defense stops hidden malicious behaviors in digital agents
Researchers developed a method to prevent dormant poisoning attacks from activating during fine-tuning.
Attackers can hide malicious behaviors that stay secret until benign fine-tuning occurs. These dormant poisoning attacks can extend to unseen actions or multiple harmful behaviors. This risk applies to language models, language agents, and multimodal agents.
The researchers created a Safety Buffer to protect against these risks. A Safety Buffer pairs harmful inputs with responses from the released model and benign versions. This method uses the fact that attackers must keep behavior safe at release to hide poisoning.
They also introduced LookAhead Defense to improve existing methods. LookAhead Defense previews benign fine-tuning updates using the Safety Buffer. It penalizes updates predicted to weaken safe behavior or increase refusals on benign versions.
LookAhead Defense reduces harmful behavior in agents from up to to at most . It reduces harmful responses in language models from up to to at most compared to vanilla fine-tuning. The method maintains competitive utility and low over-refusal.
Key facts
- Harmful behaviors can be activated by benign fine-tuning.
- LookAhead Defense reduces harmful agent behavior from up to to at most .
- The defense works across language models, language agents, and multimodal agents.