Disentangling Optimization Scale from Preference Scale in DPO

Ivan Kruzhilov

Direct Preference Optimization (DPO) is a widely used objective for aligning language models from preference data, with the coefficient $\beta$ commonly interpreted as controlling the…