KVT: k-NN Attention for Boosting Vision Transformers
arXiv:2106.00515
Abstract
Convolutional Neural Networks (CNNs) have dominated computer vision for years, due to its ability in capturing locality and translation invariance. Recently, many vision transformer architectures have been proposed and they show promising performance. A key component in vision transformers is the fully-connected self-attention which is more powerful than CNNs in modelling long range dependencies. However, since the current dense self-attention uses all image patches (tokens) to compute attention matrix, it may neglect locality of images patches and involve noisy tokens (e.g., clutter background and occlusion), leading to a slow training process and potential degradation of performance. To address these problems, we propose the -NN attention for boosting vision transformers. Specifically, instead of involving all the tokens for attention matrix calculation, we only select the top- similar tokens from the keys for each query to compute the attention map. The proposed -NN attention naturally inherits the local bias of CNNs without introducing convolutional operations, as nearby tokens tend to be more similar than others. In addition, the -NN attention allows for the exploration of long range correlation and at the same time filters out irrelevant tokens by choosing the most similar tokens from the entire image. Despite its simplicity, we verify, both theoretically and empirically, that -NN attention is powerful in speeding up training and distilling noise from input tokens. Extensive experiments are conducted by using 11 different vision transformer architectures to verify that the proposed -NN attention can work with any existing transformer architectures to improve its prediction performance. The codes are available at \url{https://github.com/damo-cv/KVT}.
Accepted by ECCV 2022
References in corpus (16)
- Transformer in Transformer
- Generating Long Sequences with Sparse Transformers
- Axial Attention in Multidimensional Transformers
- DeepViT: Towards Deeper Vision Transformer
- LocalViT: Analyzing Locality in Vision Transformers
- CvT: Introducing Convolutions to Vision Transformers
- TransReID: Transformer-based Object Re-Identification
- LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
- Sparse Sinkhorn Attention
- Rethinking Spatial Dimensions of Vision Transformers
- LambdaNetworks: Modeling Long-Range Interactions Without Attention
- Vision Transformers with Patch Diversification
- Involution: Inverting the Inherence of Convolution for Visual Recognition
- OmniNet: Omnidirectional Representations from Transformers
- Trear: Transformer-based RGB-D Egocentric Action Recognition
- TransRPPG: Remote Photoplethysmography Transformer for 3D Mask Face Presentation Attack Detection