Efficient Attention: Attention with Linear Complexities
arXiv:1812.01243
Abstract
Dot-product attention has wide applications in computer vision and natural language processing. However, its memory and computational costs grow quadratically with the input size. Such growth prohibits its application on high-resolution inputs. To remedy this drawback, this paper proposes a novel efficient attention mechanism equivalent to dot-product attention but with substantially less memory and computational costs. Its resource efficiency allows more widespread and flexible integration of attention modules into a network, which leads to better accuracies. Empirical evaluations demonstrated the effectiveness of its advantages. Efficient attention modules brought significant performance boosts to object detectors and instance segmenters on MS-COCO 2017. Further, the resource efficiency democratizes attention to complex models, where high costs prohibit the use of dot-product attention. As an exemplar, a model with efficient attention achieved state-of-the-art accuracies for stereo depth estimation on the Scene Flow dataset. Code is available at https://github.com/cmsflash/efficient-attention.
arXiv commentTo appear at WACV 2021
References in corpus (8)
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Self-Attention Generative Adversarial Networks
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond
- Cross Attention Network for Few-shot Classification
- Compact Generalized Non-local Network
- Cascade Residual Learning: A Two-stage Convolutional Neural Network for Stereo Matching
- EdgeStereo: A Context Integrated Residual Pyramid Network for Stereo Matching
Cited by in corpus (5)
- OCNet: Object Context Network for Scene Parsing
- Multi-stage Attention ResU-Net for Semantic Segmentation of Fine-Resolution Remote Sensing Images
- A Deep Generative Model for Reordering Adjacency Matrices
- Asymptotic theory of in-context learning by linear attention
- A Multi-attribute Controllable Generative Model for Histopathology Image Synthesis