Learning to summarize from human feedback

Abstract

As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about -- summary quality. In this work, we show that it is possible to significantly improve summary quality by training a model to optimize for human preferences. We collect a large, high-quality dataset of human comparisons between summaries, train a model to predict the human-preferred summary, and use that model as a reward function to fine-tune a summarization policy using reinforcement learning. We apply our method to a version of the TL;DR dataset of Reddit posts and find that our models significantly outperform both human reference summaries and much larger models fine-tuned with supervised learning alone. Our models also transfer to CNN/DM news articles, producing summaries nearly as good as the human reference without any news-specific fine-tuning. We conduct extensive analyses to understand our human feedback dataset and fine-tuned models We establish that our reward model generalizes to new datasets, and that optimizing our reward model results in better summaries than optimizing ROUGE according to humans. We hope the evidence from our paper motivates machine learning researchers to pay closer attention to how their training loss affects the model behavior they actually want.

arXiv commentNeurIPS 2020

References in corpus (32)

• Adam: A Method for Stochastic Optimization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• Teaching Machines to Read and Comprehend
• A Deep Reinforced Model for Abstractive Summarization
• The Curious Case of Neural Text Degeneration
• Sequence Level Training with Recurrent Neural Networks
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Semi-supervised Sequence Learning
• Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
• Fine-Tuning Language Models from Human Preferences
• The Natural Language Decathlon: Multitask Learning as Question Answering
• Neural Text Generation with Unlikelihood Training
• An Actor-Critic Algorithm for Sequence Prediction
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Scalable agent alignment via reward modeling: a research direction
• Get To The Point: Summarization with Pointer-Generator Networks
• ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
• ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
• Reward-rational (implicit) choice: A unifying formalism for reward learning
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• The price of debiasing automatic metrics in natural language evaluation
• Supervising strong learners by amplifying weak experts
• BanditSum: Extractive Summarization as a Contextual Bandit
• Trading Off Diversity and Quality in Natural Language Generation
• Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators
• Better Rewards Yield Better Summaries: Learning to Summarise Without References
• Finding Generalizable Evidence by Learning to Convince Q&A Models
• Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models
• Reward Learning for Efficient Reinforcement Learning in Extractive Document Summarisation

Cited by in corpus (24)

• On the Opportunities and Risks of Foundation Models
• Evaluating Large Language Models Trained on Code
• A Survey of Human-in-the-loop for Machine Learning
• Recursively Summarizing Books with Human Feedback
• Constrained Abstractive Summarization: Preserving Factual Consistency with Constrained Generation
• A General Language Assistant as a Laboratory for Alignment
• Robust Deep Reinforcement Learning for Extractive Legal Summarization
• Scaling Laws for Transfer
• Respect for Human Autonomy in Recommender Systems
• Open Problems in Cooperative AI
• Putting Humans in the Natural Language Processing Loop: A Survey
• Truthful AI: Developing and governing AI that does not lie
• Evaluating the Robustness of Collaborative Agents
• The MineRL BASALT Competition on Learning from Human Feedback
• Expressive Communication: A Common Framework for Evaluating Developments in Generative Models and Steering Interfaces
• Exploring Fluent Query Reformulations with Text-to-Text Transformers and Reinforcement Learning
• What Have We Achieved on Text Summarization?
• Current Limitations of Language Models: What You Need is Retrieval
• Hone as You Read: A Practical Type of Interactive Summarization
• Learning Natural Language Generation from Scratch
• High Quality Related Search Query Suggestions using Deep Reinforcement Learning
• Generating Self-Contained and Summary-Centric Question Answer Pairs via Differentiable Reward Imitation Learning
• Learning What To Do by Simulating the Past
• Grounding 'Grounding' in NLP