Addressing Some Limitations of Transformers with Feedback Memory

Abstract

Transformers have been successfully applied to sequential, auto-regressive tasks despite being feedforward networks. Unlike recurrent neural networks, Transformers use attention to capture temporal relations while processing input tokens in parallel. While this parallelization makes them computationally efficient, it restricts the model from fully exploiting the sequential nature of the input. The representation at a given layer can only access representations from lower layers, rather than the higher level representations already available. In this work, we propose the Feedback Transformer architecture that exposes all previous representations to all future representations, meaning the lowest representation of the current timestep is formed from the highest-level abstract representation of the past. We demonstrate on a variety of benchmarks in language modeling, machine translation, and reinforcement learning that the increased representation capacity can create small, shallow models with much stronger performance than comparable Transformers.

References in corpus (27)

• Teaching Machines to Read and Comprehend
• End-To-End Memory Networks
• Generating Long Sequences with Sparse Transformers
• Regularizing and Optimizing LSTM Language Models
• Gated Feedback Recurrent Neural Networks
• Quasi-Recurrent Neural Networks
• Reducing Transformer Depth on Demand with Structured Dropout
• Deep Equilibrium Models
• Neural GPUs Learn Algorithms
• Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Theoretical Limitations of Self-Attention in Neural Sequence Models
• An Analysis of Neural Language Modeling at Multiple Scales
• Stabilizing Transformers for Reinforcement Learning
• Get To The Point: Summarization with Pointer-Generator Networks
• Trellis Networks for Sequence Modeling
• R-Transformer: Recurrent Neural Network Enhanced Transformer
• Single Headed Attention RNN: Stop Thinking With Your Head
• On the Turing Completeness of Modern Neural Network Architectures
• Generalization through Memorization: Nearest Neighbor Language Models
• Compressive Transformers for Long-Range Sequence Modelling
• Augmenting Self-attention with Persistent Memory
• Dynamic Evaluation of Transformer Language Models
• Controllable Abstractive Summarization
• Pre-trained Language Model Representations for Language Generation
• Multi-Path Feedback Recurrent Neural Network for Scene Parsing
• The Importance of Being Recurrent for Modeling Hierarchical Structure

Cited by in corpus (7)

• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
• Not All Memories are Created Equal: Learning to Forget by Expiring
• Staircase Attention for Recurrent Processing of Sequences
• FNetAR: Mixing Tokens with Autoregressive Fourier Transforms
• Do Transformers Need Deep Long-Range Memory
• Compositional Attention: Disentangling Search and Retrieval
• E.T.: Entity-Transformers. Coreference augmented Neural Language Model for richer mention representations via Entity-Transformer blocks