SEAL: Segment-wise Extractive-Abstractive Long-form Text Summarization

Abstract

Most prior work in the sequence-to-sequence paradigm focused on datasets with input sequence lengths in the hundreds of tokens due to the computational constraints of common RNN and Transformer architectures. In this paper, we study long-form abstractive text summarization, a sequence-to-sequence setting with input sequence lengths up to 100,000 tokens and output sequence lengths up to 768 tokens. We propose SEAL, a Transformer-based model, featuring a new encoder-decoder attention that dynamically extracts/selects input snippets to sparsely attend to for each output segment. Using only the original documents and summaries, we derive proxy labels that provide weak supervision for extractive layers simultaneously with regular supervision from abstractive summaries. The SEAL model achieves state-of-the-art results on existing long-form summarization tasks, and outperforms strong baseline models on a new dataset/task we introduce, Search2Wiki, with much longer input text. Since content selection is explicit in the SEAL model, a desirable side effect is that the selection can be inspected for enhanced interpretability.

References in corpus (10)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Generating Long Sequences with Sparse Transformers
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting
• On Extractive and Abstractive Neural Document Summarization with Transformer Language Models
• Hard Mixtures of Experts for Large Scale Weakly Supervised Vision

Cited by in corpus (7)

• Recursively Summarizing Books with Human Feedback
• Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents
• Efficient Attentions for Long Document Summarization
• What's New? Summarizing Contributions in Scientific Literature
• EASE: Extractive-Abstractive Summarization with Explanations
• Bringing Structure into Summaries: a Faceted Summarization Dataset for Long Scientific Documents
• MiRANews: Dataset and Benchmarks for Multi-Resource-Assisted News Summarization