The Representation-Rationalizability Tradeoff in Reward Learning

Jing Dong, Yaoliang Yu, Pascal Pourpart

In RLHF, each training example contains a prompt $x$ and two candidate responses $y,y^{\prime }$, and annotators provide pairwise preferences between these responses. The learning problem i…