Reward Shaping to Mitigate Reward Hacking in RLHF

Jiayi Fu, Xuandong Zhao, Chengyuan Yao +4

Reinforcement learning from human feedback (RLHF) is widely used to align large language models (LLMs) with human preferences. However, RLHF remains vulnerable to \emph{reward hack…