Automatically Finding Reward Model Biases

Atticus Wang, Iván Arcuschin, Arthur Conmy

Reward models are central to large language model (LLM) post-training. However, past work has shown that they can reward spurious or undesirable attributes such as length, format,…