Confronting Reward Model Overoptimization with Constrained RLHF

Ted Moskovitz, Aaditya K. Singh, DJ Strouse +4

Large language models are typically aligned with human preferences by optimizing $\text{reward models}$ (RMs) fitted to human feedback. However, human preferences are multi-facet…