Generating Wikipedia by Summarizing Long Sequences
arXiv:1801.10198
Abstract
We show that generating English Wikipedia articles can be approached as a multi- document summarization of source documents. We use extractive summarization to coarsely identify salient information and a neural abstractive model to generate the article. For the abstractive model, we introduce a decoder-only architecture that can scalably attend to very long sequences, much longer than typical encoder- decoder architectures used in sequence transduction. We show that this model can generate fluent, coherent multi-sentence paragraphs and even whole Wikipedia articles. When given reference documents, we show it can extract relevant factual information as reflected in perplexity, ROUGE scores and human evaluations.
Published as a conference paper at ICLR 2018
References in corpus (4)
Cited by in corpus (40)
- Learning Transferable Visual Models From Natural Language Supervision
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Language Models are Few-Shot Learners
- Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Generating Long Sequences with Sparse Transformers
- Long Range Arena: A Benchmark for Efficient Transformers
- Scaling Laws for Autoregressive Generative Modeling
- Pre-training via Paraphrasing
- Graph Neural Networks for Natural Language Processing: A Survey
- Recursively Summarizing Books with Human Feedback
- Conversational Emotion Analysis via Attention Mechanisms
- Machine Reading Comprehension: a Literature Review
- ETC: Encoding Long and Structured Inputs in Transformers
- Towards a Neural Network Approach to Abstractive Multi-Document Summarization
- A General Language Assistant as a Laboratory for Alignment
- Neural Language Generation: Formulation, Methods, and Evaluation
- Read + Verify: Machine Reading Comprehension with Unanswerable Questions
- Strategies for Structuring Story Generation
- Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks
- Towards Clinical Encounter Summarization: Learning to Compose Discharge Summaries from Prior Notes
- Inductive Biases and Variable Creation in Self-Attention Mechanisms
- Shaping the Narrative Arc: An Information-Theoretic Approach to Collaborative Dialogue
- Low-Rank Bottleneck in Multi-head Attention Models
- Neural Temporal Point Processes For Modelling Electronic Health Records
- Summarize, Outline, and Elaborate: Long-Text Generation via Hierarchical Supervision from Extractive Summaries
- What Do You Mean I'm Funny? Personalizing the Joke Skill of a Voice-Controlled Virtual Assistant
- TENT: Tensorized Encoder Transformer for Temperature Forecasting
- Automated Query Reformulation for Efficient Search based on Query Logs From Stack Overflow
- Open-Domain Dialogue Generation Based on Pre-trained Language Models
- Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognition
- A Simple and Efficient Multi-Task Learning Approach for Conditioned Dialogue Generation
- Reference and Document Aware Semantic Evaluation Methods for Korean Language Summarization
- A Comprehensive Attempt to Research Statement Generation
- R2D2: Relational Text Decoding with Transformers
- Self-Attentional Models Application in Task-Oriented Dialogue Generation Systems
- INS: An Interactive Chinese News Synthesis System
- DESCGEN: A Distantly Supervised Dataset for Generating Abstractive Entity Descriptions
- Artificial Intelligence and its impact on the Fourth Industrial Revolution: A Review
- WikiAsp: A Dataset for Multi-domain Aspect-based Summarization