Masking Frequent Tokens Sharpens Direct Preference Optimization

Harshvardhan Saini, Samyak Jha, Yiming Tang +1

Direct Preference Optimization (DPO) aligns language models by optimizing over sequence-level sums of token-wise implicit reward differences. However, we identify a pervasive patho…