most citedHelping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking

3 citations · 3 across the 2 of their papers we have counts for

collaborators
Showing cs.LGShow all

2 papers · 1 filter

cs.LG2024

Verified Neural Compressed Sensing

Rudy Bunel, Krishnamurthy Dvijotham, M. Pawan Kumar +2

We develop the first (to the best of our knowledge) provably correct neural networks for a precise computational task, with the proof of correctness generated by an automated verif…

cs.LG2023★ 3 cited

Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking

Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal +9

Reward models play a key role in aligning language model applications towards human preferences. However, this setup creates an incentive for the language model to exploit errors i…