Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking

Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal +9

Reward models play a key role in aligning language model applications towards human preferences. However, this setup creates an incentive for the language model to exploit errors i…