Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization

Zhuotong Chen, Fang Liu, Jennifer Zhu +2

Direct Preference Optimization (DPO) and its variants have become the de facto standards for aligning large language models (LLMs) with human preferences or specific goals. However…