Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Abstract

We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data, efficiently improving our datasets and models. Finally, we investigate the robustness of RLHF training, and identify a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization. Alongside our main results, we perform peripheral analyses on calibration, competing objectives, and the use of OOD detection, compare our models with human writers, and provide samples from our models using prompts appearing in recent related work.

arXiv commentData available at https://github.com/anthropics/hh-rlhf

Cited by in corpus (19)

• Auditing large language models: a three-layered approach
• Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics
• "We Need Structured Output": Towards User-centered Constraints on Large Language Model Output
• Exploring the psychology of LLMs' Moral and Legal Reasoning
• A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias
• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• Learning Agent-based Modeling with LLM Companions: Experiences of Novices and Experts Using ChatGPT & NetLogo Chat
• From Screens to Scenes: A Survey of Embodied AI in Healthcare
• Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Models
• Transition Role of Entangled Data in Quantum Machine Learning
• Joint Estimation and Prediction of City-wide Delivery Demand: A Large Language Model Empowered Graph-based Learning Approach
• Reward Guidance for Reinforcement Learning Tasks Based on Large Language Models: The LMGT Framework
• Normative Conflicts and Shallow AI Alignment
• Complex QA and language models hybrid architectures, Survey
• Evaluating Language Model Agency through Negotiations
• Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval
• ArtPerception: ASCII Art-based Jailbreak on LLMs with Recognition Pre-test
• On the attribution of confidence to large language models
• Stars, Stripes, and Silicon: Unravelling the ChatGPT's All-American, Monochrome, Cis-centric Bias