Can You Trust Your Preference Models? Stackelberg Games for Alignment

Riccardo Formenti, Aditya Agrawal, Vikas Garg

Language models are frequently aligned to human preferences via imperfect evaluators whose errors can be exploited during optimization. We train policies to search for comparisons…