CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF

Licheng Pan, Haocheng Yang, Yuzhou Chen +7

Feedback noise is pervasive in preference data, arising from subjective human judgments, annotation mistakes, and imperfect LLM judges. Standard reward modeling trained on such fee…