Augmenting Self-attention with Persistent Memory

Abstract

Transformer networks have lead to important progress in language modeling and machine translation. These models include two consecutive modules, a feed-forward layer and a self-attention layer. The latter allows the network to capture long term dependencies and are often regarded as the key ingredient in the success of Transformers. Building upon this intuition, we propose a new model that solely consists of attention layers. More precisely, we augment the self-attention layers with persistent memory vectors that play a similar role as the feed-forward layer. Thanks to these vectors, we can remove the feed-forward layer without degrading the performance of a transformer. Our evaluation shows the benefits brought by our model on standard character and word level language modeling benchmarks.

References in corpus (9)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• On the difficulty of training Recurrent Neural Networks
• Recurrent Neural Network Regularization
• Exploring the Limits of Language Modeling
• End-To-End Memory Networks
• Generating Long Sequences with Sparse Transformers
• Pay Less Attention with Lightweight and Dynamic Convolutions
• An Analysis of Neural Language Modeling at Multiple Scales
• Improving Neural Language Models with a Continuous Cache

Cited by in corpus (18)

• Reformer: The Efficient Transformer
• Transformers for Modeling Physical Systems
• Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
• Training with Quantization Noise for Extreme Model Compression
• MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning
• Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving
• Addressing Some Limitations of Transformers with Feedback Memory
• Augmenting Transformers with KNN-Based Composite Memory for Dialogue
• Mask Attention Networks: Rethinking and Strengthen Transformer
• Pay Attention when Required
• DFSMN-SAN with Persistent Memory Model for Automatic Speech Recognition
• Not All Memories are Created Equal: Learning to Forget by Expiring
• Is Attention All What You Need? -- An Empirical Investigation on Convolution-Based Active Memory and Self-Attention
• Adaptive Transformers in RL
• Simplified Self-Attention for Transformer-based End-to-End Speech Recognition
• When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute
• Large Product Key Memory for Pretrained Language Models
• Lyapunov-based uncertainty-aware safe reinforcement learning