Rethinking DPO: The Role of Rejected Responses in Preference Misalignment

Jay Hyeon Cho, JunHyeok Oh, Myunsoo Kim +1

Direct Preference Optimization (DPO) is a simple and efficient framework that has attracted substantial attention. However, it often struggles to meet its primary objectives -- inc…