Orthogonal Finetuning for Direct Preference Optimization

Chenxu Yang, Ruipeng Jia, Naibin Gu +7

DPO is an effective preference optimization algorithm. However, the DPO-tuned models tend to overfit on the dispreferred samples, manifested as overly long generations lacking dive…