Fine-Tuning Language Models from Human Preferences

Abstract

Reward learning enables the application of reinforcement learning (RL) to tasks where reward is defined by human judgment, building a model of reward by asking humans questions. Most work on reward learning has used simulated environments, but complex information about values is often expressed in natural language, and we believe reward learning for language is a key to making RL practical and safe for real-world tasks. In this paper, we build on advances in generative pretraining of language models to apply reward learning to four natural language tasks: continuing text with positive sentiment or physically descriptive language, and summarization tasks on the TL;DR and CNN/Daily Mail datasets. For stylistic continuation we achieve good results with only 5,000 comparisons evaluated by humans. For summarization, models trained with 60,000 comparisons copy whole sentences from the input but skip irrelevant preamble; this leads to reasonable ROUGE scores and very good performance according to our human labelers, but may be exploiting the fact that labelers rely on simple heuristics.

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Teaching Machines to Read and Comprehend
• A Deep Reinforced Model for Abstractive Summarization
• Sequence Level Training with Recurrent Neural Networks
• Semi-supervised Sequence Learning
• Learning to Generate Reviews and Discovering Sentiment
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Scalable agent alignment via reward modeling: a research direction
• Get To The Point: Summarization with Pointer-Generator Networks
• Discriminative Active Learning
• Sample Efficient Text Summarization Using a Single Pre-Trained Transformer
• Active Learning for Speech Recognition: the Power of Gradients
• AI safety via debate
• Supervising strong learners by amplifying weak experts
• Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators
• Reward Learning for Efficient Reinforcement Learning in Extractive Document Summarisation

Cited by in corpus (63)

• Language Models are Few-Shot Learners
• Generative AI
• A Survey on Multimodal Large Language Models
• A Survey of Human-in-the-loop for Machine Learning
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Do Users Write More Insecure Code with AI Assistants?
• Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
• Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics
• Plug and Play Language Models: A Simple Approach to Controlled Text Generation
• Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
• Machine Culture
• Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators
• Recursively Summarizing Books with Human Feedback
• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• Learning to summarize from human feedback
• REFINER: Reasoning Feedback on Intermediate Representations
• A Distributional Approach to Controlled Text Generation
• Automatic Personalized Impression Generation for PET Reports Using Large Language Models
• Domain-specific ChatBots for Science using Embeddings
• Large language models for automated scholarly paper review: A survey
• Robust Deep Reinforcement Learning for Extractive Legal Summarization
• Revisiting the Plastic Surgery Hypothesis via Large Language Models
• Neural Language Generation: Formulation, Methods, and Evaluation
• Reinforcement Learning from Human Feedback: Whose Culture, Whose Values, Whose Perspectives?
• A Reliable Knowledge Processing Framework for Combustion Science using Foundation Models
• CoCon: A Self-Supervised Approach for Controlled Text Generation
• Deep Extrapolation for Attribute-Enhanced Generation
• MemoVis: A GenAI-Powered Tool for Creating Companion Reference Images for 3D Design Feedback
• Learning Norms from Stories: A Prior for Value Aligned Agents
• Unsupervised Contextual Paraphrase Generation using Lexical Control and Reinforcement Learning
• Transforming Agency. On the mode of existence of Large Language Models
• PyTester: Deep Reinforcement Learning for Text-to-Testcase Generation
• ETC-NLG: End-to-end Topic-Conditioned Natural Language Generation
• Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback
• Evaluating Models of Robust Word Recognition with Serial Reproduction
• SideControl: Controlled Open-domain Dialogue Generation via Additive Side Networks
• Sensitivity of BLANC to human-scored qualities of text summaries
• ColdGANs: Taming Language GANs with Cautious Sampling Strategies
• Heuristic-free Optimization of Force-Controlled Robot Search Strategies in Stochastic Environments
• A Multilingual Virtual Guide for Self-Attachment Technique
• Energy-Based Models for Code Generation under Compilability Constraints
• GeDi: Generative Discriminator Guided Sequence Generation
• Neural Rule-Execution Tracking Machine For Transformer-Based Text Generation
• Textual interpretation of transient image classifications from large language models
• Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval
• Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control
• MLE-guided parameter search for task loss minimization in neural sequence modeling
• Language Model Evaluation in Open-ended Text Generation
• Constrained Text Generation with Global Guidance -- Case Study on CommonGen
• Directed Beam Search: Plug-and-Play Lexically Constrained Language Generation
• Summarizing Text on Any Aspects: A Knowledge-Informed Weakly-Supervised Approach
• The Fair Game: Auditing & Debiasing AI Algorithms Over Time
• Pre-trained Language Model based Ranking in Baidu Search
• Six misconceptions about large language models: A minimal model and diagnostic taxonomy
• Generating Self-Contained and Summary-Centric Question Answer Pairs via Differentiable Reward Imitation Learning
• Automatically Exposing Problems with Neural Dialog Models
• Learning Natural Language Generation from Scratch
• CFL: Causally Fair Language Models Through Token-level Attribute Controlled Generation
• Style Control for Schema-Guided Natural Language Generation
• Calibrate your listeners! Robust communication-based training for pragmatic speakers
• Training Conversational Agents with Generative Conversational Networks
• Attribute Alignment: Controlling Text Generation from Pre-trained Language Models
• Estimating Subjective Crowd-Evaluations as an Additional Objective to Improve Natural Language Generation