When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHF

Yifan Xu, Xichen Ye, Yifan Chen +1

Quality of datasets plays an important role in large language model (LLM) alignment. In collecting human feedback, however, preference flipping is ubiquitous and causes corruption…