Luna: Linear Unified Nested Attention
arXiv:2106.01540
Abstract
The quadratic computational and memory complexities of the Transformer's attention mechanism have limited its scalability for modeling long sequences. In this paper, we propose Luna, a linear unified nested attention mechanism that approximates softmax attention with two nested linear attention functions, yielding only linear (as opposed to quadratic) time and space complexity. Specifically, with the first attention function, Luna packs the input sequence into a sequence of fixed length. Then, the packed sequence is unpacked using the second attention function. As compared to a more traditional attention mechanism, Luna introduces an additional sequence with a fixed length as input and an additional corresponding output, which allows Luna to perform attention operation linearly, while also storing adequate contextual information. We perform extensive evaluations on three benchmarks of sequence modeling tasks: long-context sequence modeling, neural machine translation and masked language modeling for large-scale pretraining. Competitive or even better experimental results demonstrate both the effectiveness and efficiency of Luna compared to a variety
arXiv commentCamera-Ready version in NeurIPS 2021. 2 figures, 9 tables
References in corpus (13)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Decoupled Weight Decay Regularization
- Longformer: The Long-Document Transformer
- Linformer: Self-Attention with Linear Complexity
- Generating Long Sequences with Sparse Transformers
- Reformer: The Efficient Transformer
- A Simple Method for Commonsense Reasoning
- Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
- Random Feature Attention
- Rethinking Attention with Performers
- On Layer Normalization in the Transformer Architecture
- GMAT: Global Memory Augmentation for Transformers
- Coordination Among Neural Modules Through a Shared Global Workspace