On the Role of Preference Variance in Preference Optimization

Jiacheng Guo, Zihao Li, Jiahao Qiu +2

Direct Preference Optimization (DPO) has emerged as an important approach for learning from human preferences in aligning large language models (LLMs). However, collecting human pr…