Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shift

Chengtao Jian, Kai Yang, Tianhao Gao +5

Direct Preference Learning has emerged as a dominant offline paradigm for preference optimization. Most of these methods are based on the Bradley-Terry (BT) model for pairwise pref…