Local-to-Global Self-Attention in Vision Transformers
arXiv:2107.04735
Abstract
Transformers have demonstrated great potential in computer vision tasks. To avoid dense computations of self-attentions in high-resolution visual data, some recent Transformer models adopt a hierarchical design, where self-attentions are only computed within local windows. This design significantly improves the efficiency but lacks global feature reasoning in early stages. In this work, we design a multi-path structure of the Transformer, which enables local-to-global reasoning at multiple granularities in each stage. The proposed framework is computationally efficient and highly effective. With a marginal increasement in computational overhead, our model achieves notable improvements in both image classification and semantic segmentation. Code is available at https://github.com/ljpadam/LG-Transformer
References in corpus (7)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Conditional Positional Encodings for Vision Transformers
- Visual Transformers: Token-based Image Representation and Processing for Computer Vision
- LocalViT: Analyzing Locality in Vision Transformers
- CvT: Introducing Convolutions to Vision Transformers
- Pre-Trained Image Processing Transformer
- MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers