Efficient Adaptation of Pretrained Transformers for Abstractive Summarization

Abstract

Large-scale learning of transformer language models has yielded improvements on a variety of natural language understanding tasks. Whether they can be effectively adapted for summarization, however, has been less explored, as the learned representations are less seamlessly integrated into existing neural text production architectures. In this work, we propose two solutions for efficiently adapting pretrained transformer language models as text summarizers: source embeddings and domain-adaptive training. We test these solutions on three abstractive summarization datasets, achieving new state of the art performance on two of them. Finally, we show that these improvements are achieved by producing more focused summaries with fewer superfluous and that performance improvements are more pronounced on more abstractive datasets.

References in corpus (11)

• Teaching Machines to Read and Comprehend
• A Deep Reinforced Model for Abstractive Summarization
• SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents
• Semi-supervised Sequence Learning
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Learned in Translation: Contextualized Word Vectors
• Neural Summarization by Extracting Sentences and Words
• Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting
• Neural Abstractive Text Summarization with Sequence-to-Sequence Models
• Abstractive Summarization of Reddit Posts with Multi-level Memory Networks
• Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization

Cited by in corpus (8)

• Extracting Training Data from Large Language Models
• Abstractive Dialog Summarization with Semantic Scaffolds
• Summary Level Training of Sentence Rewriting for Abstractive Summarization
• Classification Betters Regression in Query-based Multi-document Summarisation Techniques for Question Answering: Macquarie University at BioASQ7b
• Keyphrase Generation with Cross-Document Attention
• Nutribullets Hybrid: Multi-document Health Summarization
• Restructuring Conversations using Discourse Relations for Zero-shot Abstractive Dialogue Summarization
• Understanding Neural Abstractive Summarization Models via Uncertainty