AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization
arXiv:2103.11332
Abstract
State-of-the-art abstractive summarization models generally rely on extensive labeled data, which lowers their generalization ability on domains where such data are not available. In this paper, we present a study of domain adaptation for the abstractive summarization task across six diverse target domains in a low-resource setting. Specifically, we investigate the second phase of pre-training on large-scale generative models under three different settings: 1) source domain pre-training; 2) domain-adaptive pre-training; and 3) task-adaptive pre-training. Experiments show that the effectiveness of pre-training is correlated with the similarity between the pre-training data and the target domain task. Moreover, we find that continuing pre-training could lead to the pre-trained model's catastrophic forgetting, and a learning method with less forgetting can alleviate this issue. Furthermore, results illustrate that a huge gap still exists between the low-resource and high-resource settings, which highlights the need for more advanced domain adaptation methods for the abstractive summarization task.
arXiv commentThe first two authors contributed equally. Accepted as a long paper in NAACL 2021
References in corpus (20)
- Sequence to Sequence Learning with Neural Networks
- Overcoming catastrophic forgetting in neural networks
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Teaching Machines to Read and Comprehend
- A Deep Reinforced Model for Abstractive Summarization
- Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
- SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization
- Modeling Coverage for Neural Machine Translation
- Text Summarization with Pretrained Encoders
- Get To The Point: Summarization with Pointer-Generator Networks
- Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting
- ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
- Meta-learning for Few-shot Natural Language Processing: A Survey
- Exploring Domain Shift in Extractive Text Summarization
- CrossNER: Evaluating Cross-Domain Named Entity Recognition
- Dimsum @LaySumm 20: BART-based Approach for Scientific Document Summarization
- CAiRE-COVID: A Question Answering and Query-focused Multi-Document Summarization System for COVID-19 Scholarly Information Management
- Abstractive Summarization for Low Resource Data using Domain Transfer and Data Synthesis
- A Pilot Study of Domain Adaptation Effect for Neural Abstractive Summarization