RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
arXiv:2309.00267
Abstract
Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences, but gathering high-quality preference labels is expensive. RL from AI Feedback (RLAIF), introduced in Bai et al., offers a promising alternative that trains the reward model (RM) on preferences generated by an off-the-shelf LLM. Across the tasks of summarization, helpful dialogue generation, and harmless dialogue generation, we show that RLAIF achieves comparable performance to RLHF. Furthermore, we take a step towards "self-improvement" by demonstrating that RLAIF can outperform a supervised fine-tuned baseline even when the AI labeler is the same size as the policy, or even the exact same checkpoint as the initial policy. Finally, we introduce direct-RLAIF (d-RLAIF) - a technique that circumvents RM training by obtaining rewards directly from an off-the-shelf LLM during RL, which achieves superior performance to canonical RLAIF. Our results suggest that RLAIF can achieve performance on-par with using human feedback, offering a potential solution to the scalability limitations of RLHF.
Presented at ICML 2024
Cited by in corpus (6)
- Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
- Survey and Experiments on Mental Disorder Detection via Social Media: From Large Language Models and RAG to Agents
- Can AI Rely on the Systematicity of Truth? The Challenge of Modelling Normative Domains
- Evaluating Language Model Agency through Negotiations
- OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responses
- Towards AI epidemiology: a measurement standardisation framework for prospective risk detection