SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
arXiv:1910.00998
Abstract
We propose an end-to-end neural model for zero-shot abstractive text summarization of paragraphs, and introduce a benchmark task, ROCSumm, based on ROCStories, a subset for which we collected human summaries. In this task, five-sentence stories (paragraphs) are summarized with one sentence, using human summaries only for evaluation. We show results for extractive and human baselines to demonstrate a large abstractive gap in performance. Our model, SummAE, consists of a denoising auto-encoder that embeds sentences and paragraphs in a common space, from which either can be decoded. Summaries for paragraphs are generated by decoding a sentence from the paragraph representations. We find that traditional sequence-to-sequence auto-encoders fail to produce good summaries and describe how specific architectural choices and pre-training techniques can significantly improve performance, outperforming extractive baselines. The data, training, evaluation code, and best model weights are open-sourced.
first two authors contributed equally
References in corpus (6)
- Sequence to Sequence Learning with Neural Networks
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Categorical Reparameterization with Gumbel-Softmax
- Exploring the Limits of Language Modeling
- MASS: Masked Sequence to Sequence Pre-training for Language Generation
- A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories
Cited by in corpus (5)
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Language Models are Few-Shot Learners
- Leveraging Lead Bias for Zero-shot Abstractive News Summarization
- AgreeSum: Agreement-Oriented Multi-Document Summarization
- Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation