Gradient Imbalance in Direct Preference Optimization

Qinwei Ma, Jingzhe Shi, Can Jin +3

Direct Preference Optimization (DPO) has been proposed as a promising alternative to Proximal Policy Optimization (PPO) based Reinforcement Learning with Human Feedback (RLHF). How…