SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences

Arpan Mukherjee, Marcello Bullo, Deniz Gündüz

Uniform-reward reinforcement learning from human feedback (RLHF), which trains a single reward model to represent the preferences of all annotators, fails to capture the diversity…