Reward Model Ensembles Help Mitigate Overoptimization

Thomas Coste, Usman Anwar, Robert Kirk +1

Reinforcement learning from human feedback (RLHF) is a standard approach for fine-tuning large language models to follow instructions. As part of this process, learned reward model…