Rater State Bias in RLHF Preference Data: An Audit Framework

Elena Kopteva, Vitaliy Hlynianyi-Zhuk

We identify a structured confound in Reinforcement Learning from Human Feedback (RLHF). Pairwise preference labels are intended to reflect the compared outputs, but they may also r…