Leveraging Lead Bias for Zero-shot Abstractive News Summarization
arXiv:1912.11602
Abstract
A typical journalistic convention in news articles is to deliver the most salient information in the beginning, also known as the lead bias. While this phenomenon can be exploited in generating a summary, it has a detrimental effect on teaching a model to discriminate and extract important information in general. We propose that this lead bias can be leveraged in our favor in a simple and effective way to pre-train abstractive news summarization models on large-scale unlabeled news corpora: predicting the leading sentences using the rest of an article. We collect a massive news corpus and conduct data cleaning and filtering via statistical analysis. We then apply self-supervised pre-training on this dataset to existing generation models BART and T5 for domain adaptation. Via extensive experiments on six benchmark datasets, we show that this approach can dramatically improve the summarization quality and achieve state-of-the-art results for zero-shot news summarization without any fine-tuning. For example, in the DUC2003 dataset, the ROUGE-1 score of BART increases 13.7% after the lead-bias pre-training. We deploy the model in Microsoft News and provide public APIs as well as a demo website for multi-lingual news summarization.
arXiv commentPublished in ACM SIGIR 2021
References in corpus (12)
- A Deep Reinforced Model for Abstractive Summarization
- Unified Language Model Pre-training for Natural Language Understanding and Generation
- On the Variance of the Adaptive Learning Rate and Beyond
- Passage Re-ranking with BERT
- SDNet: Contextualized Attention-based Deep Network for Conversational Question Answering
- MeanSum: A Neural Model for Unsupervised Multi-document Abstractive Summarization
- Learning-Based Single-Document Summarization with Compression and Anaphoricity Constraints
- SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
- Enhancing Factual Consistency of Abstractive Summarization
- TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
- Importance of Copying Mechanism for News Headline Generation
- Pre-training for Abstractive Document Summarization by Reinstating Source Text