Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment

Yan Liu, Xiaoyuan Yi, Xiaokang Chen +6

The demand for regulating potentially risky behaviors of large language models (LLMs) has ignited research on alignment methods. Since LLM alignment heavily relies on reward models…