Neural Abstractive Text Summarization with Sequence-to-Sequence Models

Abstract

In the past few years, neural abstractive text summarization with sequence-to-sequence (seq2seq) models have gained a lot of popularity. Many interesting techniques have been proposed to improve seq2seq models, making them capable of handling different challenges, such as saliency, fluency and human readability, and generate high-quality summaries. Generally speaking, most of these techniques differ in one of these three categories: network structure, parameter inference, and decoding/generation. There are also other concerns, such as efficiency and parallelism for training a model. In this paper, we provide a comprehensive literature survey on different seq2seq models for abstractive text summarization from the viewpoint of network structures, training strategies, and summary generation algorithms. Several models were first proposed for language modeling and generation tasks, such as machine translation, and later applied to abstractive text summarization. Hence, we also provide a brief review of these models. As part of this survey, we also develop an open source library, namely, Neural Abstractive Text Summarizer (NATS) toolkit, for the abstractive text summarization. An extensive set of experiments have been conducted on the widely used CNN/Daily Mail dataset to examine the effectiveness of several different neural network components. Finally, we benchmark two models implemented in NATS on the two recently released datasets, namely, Newsroom and Bytecup.

References in corpus (40)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• On the difficulty of training Recurrent Neural Networks
• WaveNet: A Generative Model for Raw Audio
• Teaching Machines to Read and Comprehend
• Convolutional Sequence to Sequence Learning
• BERTScore: Evaluating Text Generation with BERT
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Tutorial on Variational Autoencoders
• A Deep Reinforced Model for Abstractive Summarization
• The Curious Case of Neural Text Degeneration
• PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• Sequence Level Training with Recurrent Neural Networks
• SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents
• A Recurrent Latent Variable Model for Sequential Data
• Deep Reinforcement Learning for Dialogue Generation
• The Natural Language Decathlon: Multitask Learning as Question Answering
• Quasi-Recurrent Neural Networks
• Neural Machine Translation in Linear Time
• Selective Encoding for Abstractive Sentence Summarization
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• An Actor-Critic Algorithm for Sequence Prediction
• WikiHow: A Large Scale Text Summarization Dataset
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• The Optimal Reward Baseline for Gradient-Based Reinforcement Learning
• Reinforcement Learning Neural Turing Machines - Revised
• Generating News Headlines with Recurrent Neural Networks
• Mutual Information and Diverse Decoding Improve Neural Machine Translation
• Structure-Infused Copy Mechanisms for Abstractive Summarization
• Efficient Summarization with Read-Again and Copy Mechanism
• Temporal Attention Model for Neural Machine Translation
• Actor-Critic based Training Framework for Abstractive Summarization
• An Introductory Survey on Attention Mechanisms in NLP Problems
• Generative Adversarial Network for Abstractive Text Summarization
• Learning to Extract Coherent Summary via Deep Reinforcement Learning
• Strongly-Typed Recurrent Neural Networks
• Diverse Beam Search for Increased Novelty in Abstractive Summarization

Cited by in corpus (14)

• Medical Deep Learning -- A systematic Meta-Review
• Evaluation of Text Generation: A Survey
• Natural Language Processing Advancements By Deep Learning: A Survey
• Deep Learning for Sequential Recommendation: Algorithms, Influential Factors, and Evaluations
• Efficient Adaptation of Pretrained Transformers for Abstractive Summarization
• Text-to-SQL Generation for Question Answering on Electronic Medical Records
• Efficiency Metrics for Data-Driven Models: A Text Summarization Case Study
• Abstractive Text Summarization based on Language Model Conditioning and Locality Modeling
• Towards Automated Anamnesis Summarization: BERT-based Models for Symptom Extraction
• LeafNATS: An Open-Source Toolkit and Live Demo System for Neural Abstractive Text Summarization
• Discriminative Pre-training for Low Resource Title Compression in Conversational Grocery
• Using Natural Language Processing to Understand Reasons and Motivators Behind Customer Calls in Financial Domain
• Deep Learning -- A first Meta-Survey of selected Reviews across Scientific Disciplines, their Commonalities, Challenges and Research Impact
• Deep Reinforced Self-Attention Masks for Abstractive Summarization (DR.SAS)