A Unifying Lens on Reward Uncertainty in RLHF

Ely Hahami, Yoel Zimmermann, Ray Zhou +1

Reinforcement learning from human feedback (RLHF) is bottlenecked by reward hacking, where the policy exploits errors in a proxy reward model (RM) and produces high RM scores witho…