RLHF May Not Reflect Genuine Preferences

Bijean Ghafouri, Eun Cheol Choi, Priyanka Dey +1

Reinforcement Learning from Human Feedback (RLHF) assumes that annotation responses reflect genuine human preferences. They often do not. Behavioral scientists have documented for…