Learning Optimal Advantage from Preferences and Mistaking it for Reward

W. Bradley Knox, Stephane Hatgis-Kessell, Sigurdur Orn Adalgeirsson +4

We consider algorithms for learning reward functions from human preferences over pairs of trajectory segments, as used in reinforcement learning from human feedback (RLHF). Most re…