Mitigating Mismatch within Reference-based Preference Optimization

Suqin Yuan, Xingrui Yu, Jiyang Zheng +4

Direct Preference Optimization (DPO) has become the de facto standard for offline preference alignment of large language models, but its reliance on a reference policy introduces a…