Is Attention All What You Need? -- An Empirical Investigation on Convolution-Based Active Memory and Self-Attention
arXiv:1912.11959
Abstract
The key to a Transformer model is the self-attention mechanism, which allows the model to analyze an entire sequence in a computationally efficient manner. Recent work has suggested the possibility that general attention mechanisms used by RNNs could be replaced by active-memory mechanisms. In this work, we evaluate whether various active-memory mechanisms could replace self-attention in a Transformer. Our experiments suggest that active-memory alone achieves comparable results to the self-attention mechanism for language modelling, but optimal results are mostly achieved by using both active-memory and self-attention mechanisms together. We also note that, for some specific algorithmic tasks, active-memory mechanisms alone outperform both self-attention and a combination of the two.
arXiv comment7 pages, 2 figures
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
- Convolutional Sequence to Sequence Learning
- Language Modeling with Gated Convolutional Networks
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Pointer Sentinel Mixture Models
- The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches
- Pay Less Attention with Lightweight and Dynamic Convolutions
- Neural GPUs Learn Algorithms
- R-Transformer: Recurrent Neural Network Enhanced Transformer
- Augmenting Self-attention with Persistent Memory
- Improving the Neural GPU Architecture for Algorithm Learning
- Can Active Memory Replace Attention?