UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking

Lingling Fu, Yongfu Xue

Reward models (RMs) are critical to reinforcement learning from human feedback (RLHF). However, policy models can exploit biases or spurious correlations in dense RMs, causing rewa…