MSG-Transformer: Exchanging Local Spatial Information by Manipulating Messenger Tokens
arXiv:2105.15168
Abstract
Transformers have offered a new methodology of designing neural networks for visual recognition. Compared to convolutional networks, Transformers enjoy the ability of referring to global features at each stage, yet the attention module brings higher computational overhead that obstructs the application of Transformers to process high-resolution visual data. This paper aims to alleviate the conflict between efficiency and flexibility, for which we propose a specialized token for each region that serves as a messenger (MSG). Hence, by manipulating these MSG tokens, one can flexibly exchange visual information across regions and the computational complexity is reduced. We then integrate the MSG token into a multi-scale architecture named MSG-Transformer. In standard image classification and object detection, MSG-Transformer achieves competitive performance and the inference on both GPU and CPU is accelerated. Code is available at https://github.com/hustvl/MSG-Transformer.
Accepted by CVPR 2022
References in corpus (7)
- Rethinking Atrous Convolution for Semantic Image Segmentation
- Transformer in Transformer
- MMDetection: Open MMLab Detection Toolbox and Benchmark
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
- CvT: Introducing Convolutions to Vision Transformers
- Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer
- CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows