Explaining and Preventing Alignment Collapse in Iterative RLHF

Etienne Gauthier, Francis Bach, Michael I. Jordan

Reinforcement learning from human feedback (RLHF) typically assumes a static or non-strategic reward model (RM). In iterative deployment, however, the policy generates the data on…