Pretrain Value, Not Reward: Decoupled Value Policy Optimization

Chenghua Huang, Lu Wang, Fangkai Yang +6

In this paper, we explore how directly pretraining a value model simplifies and stabilizes reinforcement learning from human feedback (RLHF). In reinforcement learning, value estim…