Fooling Reward Models with Gibberish: Adversarial Examples Beyond the Semantic Regime

Yuheng Zhang, Mingyue Huo, Minghao Zhu +2

Reward models (RMs) are widely used as optimization targets in reinforcement learning from human feedback (RLHF). We show that leading RMs can assign higher scores to gibberish tha…