Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modeling

Qiyuan Deng, Xuefeng Bai, Kehai Chen +3

Reinforcement Learning (RL) algorithms for safety alignment of Large Language Models (LLMs), such as Direct Preference Optimization (DPO), encounter the challenge of distribution s…