Preference Optimization via Contrastive Divergence: Your Reward Model is Secretly an NLL Estimator

Zhuotong Chen, Fang Liu, Xuan Zhu +2

Existing studies on preference optimization (PO) have centered on constructing pairwise preference data following simple heuristics, such as maximizing the margin between preferred…