Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?

Chia-Hsuan Lee, Mingyang Zhou, Renkun Ni +6

Preference optimization methods such as DPO and KTO are widely used for aligning language models, yet little is understood about what properties of preference data drive downstream…