ResT: An Efficient Transformer for Visual Recognition

Abstract

This paper presents an efficient multi-scale vision Transformer, called ResT, that capably served as a general-purpose backbone for image recognition. Unlike existing Transformer methods, which employ standard Transformer blocks to tackle raw images with a fixed resolution, our ResT have several advantages: (1) A memory-efficient multi-head self-attention is built, which compresses the memory by a simple depth-wise convolution, and projects the interaction across the attention-heads dimension while keeping the diversity ability of multi-heads; (2) Position encoding is constructed as spatial attention, which is more flexible and can tackle with input images of arbitrary size without interpolation or fine-tune; (3) Instead of the straightforward tokenization at the beginning of each stage, we design the patch embedding as a stack of overlapping convolution operation with stride on the 2D-reshaped token map. We comprehensively validate ResT on image classification and downstream tasks. Experimental results show that the proposed ResT can outperform the recently state-of-the-art backbones by a large margin, demonstrating the potential of ResT as strong backbones. The code and models will be made publicly available at https://github.com/wofmanaf/ResT.

arXiv commentResT is an efficient multi-scale vision Transformer that can tackle input images with arbitrary size. arXiv admin note: text overlap with arXiv:2103.14030 by other authors

References in corpus (11)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Decoupled Weight Decay Regularization
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Layer Normalization
• Pre-Trained Image Processing Transformer
• How Much Position Information Do Convolutional Neural Networks Encode?
• Multiscale Vision Transformers
• FCOS: A simple and strong anchor-free object detector
• Group-CAM: Group Score-Weighted Visual Explanations for Deep Convolutional Networks

Cited by in corpus (14)

• Transformers in Vision: A Survey
• A survey of the Vision Transformers and their CNN-Transformer based Variants
• A Novel Transformer Based Semantic Segmentation Scheme for Fine-Resolution Remote Sensing Images
• An Empirical Study of Remote Sensing Pretraining
• SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing
• A Survey of Visual Transformers
• Container: Context Aggregation Network
• Dual-stream Network for Visual Recognition
• PSLT: A Light-weight Vision Transformer with Ladder Self-Attention and Progressive Shift
• ProTo: Program-Guided Transformer for Program-Guided Tasks
• Long-term Leap Attention, Short-term Periodic Shift for Video Classification
• Transformer Meets Convolution: A Bilateral Awareness Network for Semantic Segmentation of Very Fine Resolution Urban Scene Images
• A Novel Stochastic Transformer-based Approach for Post-Traumatic Stress Disorder Detection using Audio Recording of Clinical Interviews
• Trans4Trans: Efficient Transformer for Transparent Object Segmentation to Help Visually Impaired People Navigate in the Real World