Small-Margin Preferences Still Matter-If You Train Them Right

Jinlong Pang, Zhaowei Zhu, Na Di +4

Preference optimization methods such as DPO align large language models (LLMs) using paired comparisons, but their effectiveness can be highly sensitive to the quality and difficul…