SEAL: Systematic Error Analysis for Value ALignment

Manon Revel, Matteo Cargnelutti, Tyna Eloundou +1

Reinforcement Learning from Human Feedback (RLHF) aims to align language models (LMs) with human values by training reward models (RMs) on binary preferences and using these RMs to…