Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier

Anirudhan Badrinath, Prabhat Agarwal, Jiajing Xu

For aligning large language models (LLMs), prior work has leveraged reinforcement learning via human feedback (RLHF) or variations of direct preference optimization (DPO). While DP…