Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection

Abstract

Self-attention based Transformer has demonstrated the state-of-the-art performances in a number of natural language processing tasks. Self-attention is able to model long-term dependencies, but it may suffer from the extraction of irrelevant information in the context. To tackle the problem, we propose a novel model called \textbf{Explicit Sparse Transformer}. Explicit Sparse Transformer is able to improve the concentration of attention on the global context through an explicit selection of the most relevant segments. Extensive experimental results on a series of natural language processing and computer vision tasks, including neural machine translation, image captioning, and language modeling, all demonstrate the advantages of Explicit Sparse Transformer in model performance. We also show that our proposed sparse attention method achieves comparable or better results than the previous sparse attention method, but significantly reduces training and testing time. For example, the inference speed is twice that of sparsemax in Transformer model. Code will be available at \url{https://github.com/lancopku/Explicit-Sparse-Transformer}

References in corpus (15)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Generating Long Sequences with Sparse Transformers
• Gated Feedback Recurrent Neural Networks
• Universal Transformers
• Hierarchical Multiscale Recurrent Neural Networks
• Fixup Initialization: Residual Learning Without Normalization
• Multiplicative LSTM for sequence modelling
• Latent Alignment and Variational Attention
• Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes
• MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning
• Towards Neural Phrase-based Machine Translation
• Joint Source-Target Self Attention with Locality Constraints
• Neural Phrase-to-Phrase Machine Translation
• Image Captioning as Neural Machine Translation Task in SOCKEYE

Cited by in corpus (6)

• Distance-Aware Occlusion Detection with Focused Attention
• $O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers
• MedGPT: Medical Concept Prediction from Clinical Narratives
• Untangling tradeoffs between recurrence and self-attention in neural networks
• Sparse Attention with Linear Units
• Query-Key Normalization for Transformers