RegionViT: Regional-to-Local Attention for Vision Transformers

Abstract

Vision transformer (ViT) has recently shown its strong capability in achieving comparable results to convolutional neural networks (CNNs) on image classification. However, vanilla ViT simply inherits the same architecture from the natural language processing directly, which is often not optimized for vision applications. Motivated by this, in this paper, we propose a new architecture that adopts the pyramid structure and employ a novel regional-to-local attention rather than global self-attention in vision transformers. More specifically, our model first generates regional tokens and local tokens from an image with different patch sizes, where each regional token is associated with a set of local tokens based on the spatial location. The regional-to-local attention includes two steps: first, the regional self-attention extract global information among all regional tokens and then the local self-attention exchanges the information among one regional token and the associated local tokens via self-attention. Therefore, even though local self-attention confines the scope in a local region but it can still receive global information. Extensive experiments on four vision tasks, including image classification, object and keypoint detection, semantics segmentation and action recognition, show that our approach outperforms or is on par with state-of-the-art ViT variants including many concurrent works. Our source codes and models are available at https://github.com/ibm/regionvit.

add more results and link to codes and models. https://github.com/ibm/regionvit, formatted with ICLR style

References in corpus (10)

• The Kinetics Human Action Video Dataset
• Is Space-Time Attention All You Need for Video Understanding?
• Transformer in Transformer
• Conditional Positional Encodings for Vision Transformers
• LocalViT: Analyzing Locality in Vision Transformers
• CvT: Introducing Convolutions to Vision Transformers
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• Rethinking Spatial Dimensions of Vision Transformers
• ConTNet: Why not use convolution and transformer at the same time?
• LambdaNetworks: Modeling Long-Range Interactions Without Attention

Cited by in corpus (6)

• A Survey on Visual Transformer
• Transformers in Vision: A Survey
• Vision Transformers for Single Image Dehazing
• Human Action Recognition from Various Data Modalities: A Review
• Vision Transformers: From Semantic Segmentation to Dense Prediction
• Global Interaction Modelling in Vision Transformer via Super Tokens