GOPO: Policy Optimization using Ranked Rewards

Kyuseong Choi, Dwaipayan Saha, Woojeong Kim +2

Standard reinforcement learning from human feedback (RLHF) trains a reward model on pairwise preference data and then uses it for policy optimization. However, while reward models…