Is Attention All What You Need? -- An Empirical Investigation on Convolution-Based Active Memory and Self-Attention

Thomas Dowdell, Hongyu Zhang

The key to a Transformer model is the self-attention mechanism, which allows the model to analyze an entire sequence in a computationally efficient manner. Recent work has suggeste…