Recursively Summarizing Books with Human Feedback
arXiv:2109.10862
Abstract
A major challenge for scaling machine learning is training models to perform tasks that are very difficult or time-consuming for humans to evaluate. We present progress on this problem on the task of abstractive summarization of entire fiction novels. Our method combines learning from human feedback with recursive task decomposition: we use models trained on smaller parts of the task to assist humans in giving feedback on the broader task. We collect a large volume of demonstrations and comparisons from human labelers, and fine-tune GPT-3 using behavioral cloning and reward modeling to do summarization recursively. At inference time, the model first summarizes small sections of the book and then recursively summarizes these summaries to produce a summary of the entire book. Our human labelers are able to supervise and evaluate the models quickly, despite not having read the entire books themselves. Our resulting model generates sensible summaries of entire books, even matching the quality of human-written summaries in a few cases ( of books). We achieve state-of-the-art results on the recent BookSum dataset for book-length summarization. A zero-shot question-answering model using these summaries achieves state-of-the-art results on the challenging NarrativeQA benchmark for answering questions about books and movie scripts. We release datasets of samples from our model.
References in corpus (33)
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Language Models are Few-Shot Learners
- BERTScore: Evaluating Text Generation with BERT
- Fine-Tuning Language Models from Human Preferences
- An Actor-Critic Algorithm for Sequence Prediction
- Thinking Fast and Slow with Deep Learning and Tree Search
- Hierarchical Neural Story Generation
- Text Summarization with Pretrained Encoders
- Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
- Scalable agent alignment via reward modeling: a research direction
- Generating Wikipedia by Summarizing Long Sequences
- HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization
- Distilling Knowledge from Reader to Retriever for Question Answering
- Learning to summarize from human feedback
- Hierarchical Transformers for Multi-Document Summarization
- Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
- AI safety via debate
- Supervising strong learners by amplifying weak experts
- On Extractive and Abstractive Neural Document Summarization with Transformer Language Models
- BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization
- New Alignment Methods for Discriminative Book Summarization
- SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization
- Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators
- SEAL: Segment-wise Extractive-Abstractive Long-form Text Summarization
- Answers Unite! Unsupervised Metrics for Reinforced Summarization Models
- Better Rewards Yield Better Summaries: Learning to Summarise Without References
- Hidden Incentives for Auto-Induced Distributional Shift
- Finding Generalizable Evidence by Learning to Convince Q&A Models
- Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models
- On Generating Extended Summaries of Long Documents
- ReadTwice: Reading Very Large Documents with Memories
- Extractive Summarization of Long Documents by Combining Global and Local Context
- Frustratingly Hard Evidence Retrieval for QA Over Books
Cited by in corpus (7)
- Finetuned Language Models Are Zero-Shot Learners
- ConceptEVA: Concept-Based Interactive Exploration and Customization of Document Summaries
- Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents
- Translating Legalese: Enhancing Public Understanding of Court Opinions with Legal Summarizers
- Advances in Preference-based Reinforcement Learning: A Review
- Momentum Contrastive Pre-training for Question Answering
- AI Alignment vs. AI Ethical Treatment: 10 Challenges