Procedural Fairness Failures in RLHF from Preference Averaging

M P V S Gopinadh, Karthik Kamuju, Kummari Avinash +2

Reinforcement Learning from Human Feedback (RLHF) aggregates heterogeneous preferences into a single reward model, assuming preference homogeneity. When preferences are heterogeneo…