Random Feature Attention

Abstract

Transformers are state-of-the-art models for a variety of sequence modeling tasks. At their core is an attention function which models pairwise interactions between the inputs at every timestep. While attention is powerful, it does not scale efficiently to long sequences due to its quadratic time and space complexity in the sequence length. We propose RFA, a linear time and space attention that uses random feature methods to approximate the softmax function, and explore its application in transformers. RFA can be used as a drop-in replacement for conventional softmax attention and offers a straightforward way of learning with recency bias through an optional gating mechanism. Experiments on language modeling and machine translation demonstrate that RFA achieves similar or better performance compared to strong transformer baselines. In the machine translation experiment, RFA decodes twice as fast as a vanilla transformer. Compared to existing efficient transformer variants, RFA is competitive in terms of both accuracy and efficiency on three long text classification datasets. Our analysis shows that RFA's efficiency gains are especially notable on long sequences, suggesting that RFA will be particularly useful in tasks that require working with large inputs, fast decoding speed, or low memory footprints.

ICLR 2021

References in corpus (17)

• Auto-Encoding Variational Bayes
• Distilling the Knowledge in a Neural Network
• Language Models are Few-Shot Learners
• Longformer: The Long-Document Transformer
• Linformer: Self-Attention with Linear Complexity
• Generating Long Sequences with Sparse Transformers
• Axial Attention in Multidimensional Transformers
• Reformer: The Efficient Transformer
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• Big Bird: Transformers for Longer Sequences
• Synthesizer: Rethinking Self-Attention in Transformer Models
• Using Fast Weights to Attend to the Recent Past
• Transformers with convolutional context for ASR
• Lite Transformer with Long-Short Range Attention
• Rethinking Attention with Performers
• Orthogonal Random Features

Cited by in corpus (30)

• Transformers in Vision: A Survey
• CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
• Perceiver: General Perception with Iterative Attention
• Graph Neural Networks for Natural Language Processing: A Survey
• SOFT: Softmax-free Transformer with Linear Complexity
• Choose a Transformer: Fourier or Galerkin
• Long-Short Transformer: Efficient Transformers for Language and Vision
• T-former: An Efficient Transformer for Image Inpainting
• Luna: Linear Unified Nested Attention
• An Attention Free Transformer
• Combiner: Full Attention Transformer with Sparse Computation Cost
• Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond
• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
• The NLP Task Effectiveness of Long-Range Transformers
• Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator
• Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
• AGBoost: Attention-based Modification of Gradient Boosting Machine
• FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attention
• PoNet: Pooling Network for Efficient Token Mixing in Long Sequences
• Sparse Factorization of Large Square Matrices
• Towards a Unified Quadrature Framework for Large-Scale Kernel Machines
• Unlocking Pixels for Reinforcement Learning via Implicit Attention
• Token Pooling in Vision Transformers
• $\infty$-former: Infinite Memory Transformer
• When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute
• Leveraging redundancy in attention with Reuse Transformers
• Towards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLU
• Reservoir Transformers
• LightSeq2: Accelerated Training for Transformer-based Models on GPUs
• PermuteFormer: Efficient Relative Position Encoding for Long Sequences