What do Reward Models Memorize?

Ivo Verhoeven, Pushkar Mishra, Ekaterina Shutova

This paper studies what discriminatively trained reward models (RMs) memorize by measuring counterfactual memorization on two human preference datasets. We show that RMs 1) misallo…