Towards Understanding the Influence of Reward Margin on Preference Model Performance

Bowen Qin, Duanyu Feng, Xi Yang

Reinforcement Learning from Human Feedback (RLHF) is a widely used framework for the training of language models. However, the process of using RLHF to develop a language model tha…