Fastformer: Additive Attention Can Be All You Need

Abstract

Transformer is a powerful model for text understanding. However, it is inefficient due to its quadratic complexity to input sequence length. Although there are many methods on Transformer acceleration, they are still either inefficient on long sequences or not effective enough. In this paper, we propose Fastformer, which is an efficient Transformer model based on additive attention. In Fastformer, instead of modeling the pair-wise interactions between tokens, we first use additive attention mechanism to model global contexts, and then further transform each token representation based on its interaction with global context representations. In this way, Fastformer can achieve effective context modeling with linear complexity. Extensive experiments on five datasets show that Fastformer is much more efficient than many existing Transformer models and can meanwhile achieve comparable or even better long text modeling performance.

arXiv commentAdd results on Bing Ad CVR prediction

References in corpus (9)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Teaching Machines to Read and Comprehend
• Longformer: The Long-Document Transformer
• Linformer: Self-Attention with Linear Complexity
• Generating Long Sequences with Sparse Transformers
• Reformer: The Efficient Transformer
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers
• Poolingformer: Long Document Modeling with Pooling Attention

Cited by in corpus (3)

• GRAM: Fast Fine-tuning of Pre-trained Language Models for Content-based Collaborative Filtering
• PoNet: Pooling Network for Efficient Token Mixing in Long Sequences
• Audiomer: A Convolutional Transformer For Keyword Spotting