Normalized Rewards for Preference Optimization

Shawn Im, Federico Danieli, Skyler Seto +2

Direct Alignment Algorithms (DAAs) such as DPO have become a common way to post-train and align LLMs with human preferences. However, DAAs have been observed to over-optimize their…