Rent-Seeking Language Models: Reward Models Set the Prices, Frequent Audits Curb the Rent

Zixiao Wang, Xingbo Du, Duzhen Zhang +1

Reward model overoptimization in RLHF is usually described as reward hacking. We recast it as rent-seeking: the policy, a firm, earns reward from a regulator, the proxy reward mode…