When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shift

Khoi Le, Tri Cao, Phong Nguyen +5

Weak-to-strong (W2S) generalization is a promising framework for scalable oversight, yet existing evaluations often test students under matched train-test distributions. Therefore,…