Self-Supervised Learning for Contextualized Extractive Summarization
arXiv:1906.04466
Abstract
Existing models for extractive summarization are usually trained from scratch with a cross-entropy loss, which does not explicitly capture the global context at the document level. In this paper, we aim to improve this task by introducing three auxiliary pre-training tasks that learn to capture the document-level context in a self-supervised fashion. Experiments on the widely-used CNN/DM dataset validate the effectiveness of the proposed auxiliary tasks. Furthermore, we show that after pre-training, a clean model with simple building blocks is able to outperform previous state-of-the-art that are carefully designed.
Accepted to ACL 2019
References in corpus (4)
- Teaching Machines to Read and Comprehend
- SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents
- Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
- A Hierarchical Structured Self-Attentive Model for Extractive Document Summarization (HSSAS)