Testing the Limits of Jailbreaking Defenses with the Purple Problem

Taeyoun Kim, Suhas Kotha, Aditi Raghunathan

The rise of "jailbreak" attacks on language models has led to a flurry of defenses aimed at preventing undesirable responses. We critically examine the two stages of the defense pi…