Variance-aware Reward Modeling with Anchor Guidance

Shuxing Fang, Ruijian Han, Liangyu Zhang +1

Standard Bradley--Terry (BT) reward models are limited when human preferences are pluralistic. Although soft preference labels preserve disagreement information, BT can only expres…