MSG-Transformer: Exchanging Local Spatial Information by Manipulating Messenger Tokens
arXiv:2105.15168
Abstract
Transformers have offered a new methodology of designing neural networks for visual recognition. Compared to convolutional networks, Transformers enjoy the ability of referring to global features at each stage, yet the attention module brings higher computational overhead that obstructs the application of Transformers to process high-resolution visual data. This paper aims to alleviate the conflict between efficiency and flexibility, for which we propose a specialized token for each region that serves as a messenger (MSG). Hence, by manipulating these MSG tokens, one can flexibly exchange visual information across regions and the computational complexity is reduced. We then integrate the MSG token into a multi-scale architecture named MSG-Transformer. In standard image classification and object detection, MSG-Transformer achieves competitive performance and the inference on both GPU and CPU is accelerated. Code is available at https://github.com/hustvl/MSG-Transformer.
arXiv commentAccepted by CVPR 2022
References in corpus (12)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Rethinking Atrous Convolution for Semantic Image Segmentation
- Transformer in Transformer
- MMDetection: Open MMLab Detection Toolbox and Benchmark
- DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
- CvT: Introducing Convolutions to Vision Transformers
- Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer
- CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
- FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
- Training-Time-Friendly Network for Real-Time Object Detection