Attention Head Masking for Inference Time Content Selection in Abstractive Summarization

Abstract

How can we effectively inform content selection in Transformer-based abstractive summarization models? In this work, we present a simple-yet-effective attention head masking technique, which is applied on encoder-decoder attentions to pinpoint salient content at inference time. Using attention head masking, we are able to reveal the relation between encoder-decoder attentions and content selection behaviors of summarization models. We then demonstrate its effectiveness on three document summarization datasets based on both in-domain and cross-domain settings. Importantly, our models outperform prior state-of-the-art models on CNN/Daily Mail and New York Times datasets. Moreover, our inference-time masking technique is also data-efficient, requiring only 20% of the training samples to outperform BART fine-tuned on the full CNN/DailyMail dataset.

Accepted at NAACL 2021 (short paper)

References in corpus (7)

• Teaching Machines to Read and Comprehend
• A Deep Reinforced Model for Abstractive Summarization
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Selective Encoding for Abstractive Sentence Summarization
• ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
• Attention is Not Only a Weight: Analyzing Transformers with Vector Norms
• Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Reward