Not All Preferences Deserve Gradients: Understanding Gradient Utility in Offline Reasoning Alignment

Hui Wu, Hengyi Cai, Jinman Zhao +6

Offline preference optimization aligns reasoning models from fixed chosen--rejected pairs, yet standard methods apply gradient updates from every pair regardless of its training va…