Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective

Duanyu Feng, Bowen Qin, Chen Huang +2

Direct Preference Optimization (DPO), which derives reward signals directly from pairwise preference data, has shown its effectiveness on aligning Large Language Models (LLMs) with…