Attention Optimization for Abstractive Document Summarization
arXiv:1910.11491 · doi:10.18653/v1/D19-1117
Abstract
Attention plays a key role in the improvement of sequence-to-sequence-based document summarization models. To obtain a powerful attention helping with reproducing the most salient information and avoiding repetitions, we augment the vanilla attention model from both local and global aspects. We propose an attention refinement unit paired with local variance loss to impose supervision on the attention model at each decoding step, and a global variance loss to optimize the attention distributions of all decoding steps from the global perspective. The performances on the CNN/Daily Mail dataset verify the effectiveness of our methods.
References in corpus (6)
- Neural Machine Translation by Jointly Learning to Align and Translate
- Sequence to Sequence Learning with Neural Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Teaching Machines to Read and Comprehend
- A Deep Reinforced Model for Abstractive Summarization
- SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents