Twins: Revisiting the Design of Spatial Attention in Vision Transformers

Abstract

Very recently, a variety of vision transformer architectures for dense prediction tasks have been proposed and they show that the design of spatial attention is critical to their success in these tasks. In this work, we revisit the design of the spatial attention and demonstrate that a carefully-devised yet simple spatial attention mechanism performs favourably against the state-of-the-art schemes. As a result, we propose two vision transformer architectures, namely, Twins-PCPVT and Twins-SVT. Our proposed architectures are highly-efficient and easy to implement, only involving matrix multiplications that are highly optimized in modern deep learning frameworks. More importantly, the proposed architectures achieve excellent performance on a wide range of visual tasks, including image level classification as well as dense detection and segmentation. The simplicity and strong performance suggest that our proposed architectures may serve as stronger backbones for many vision tasks. Our code is released at https://github.com/Meituan-AutoML/Twins .

arXiv commentAccepted to NeurIPS2021

References in corpus (8)

• Decoupled Weight Decay Regularization
• ParseNet: Looking Wider to See Better
• Transformer in Transformer
• OCNet: Object Context Network for Scene Parsing
• Conditional Positional Encodings for Vision Transformers
• Rigid-Motion Scattering for Texture Classification
• CvT: Introducing Convolutions to Vision Transformers
• Pre-Trained Image Processing Transformer

Cited by in corpus (20)

• Transformers in Vision: A Survey
• PVT v2: Improved Baselines with Pyramid Vision Transformer
• MedViT: A Robust Vision Transformer for Generalized Medical Image Classification
• HIPA: Hierarchical Patch Transformer for Single Image Super Resolution
• Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets
• Dual-stream Network for Visual Recognition
• Medical Image Classification with KAN-Integrated Transformers and Dilated Neighborhood Attention
• PSLT: A Light-weight Vision Transformer with Ladder Self-Attention and Progressive Shift
• MASK-CNN-Transformer For Real-Time Multi-Label Weather Recognition
• Exploring Advances in Transformers and CNN for Skin Lesion Diagnosis on Small Datasets
• Vision Transformers: From Semantic Segmentation to Dense Prediction
• Graph-Segmenter: Graph Transformer with Boundary-aware Attention for Semantic Segmentation
• UniNeXt: Exploring A Unified Architecture for Vision Recognition
• $S^2$Net: Accurate Panorama Depth Estimation on Spherical Surface
• SparX: A Sparse Cross-Layer Connection Mechanism for Hierarchical Vision Mamba and Transformer Networks
• Softmax-free Linear Transformers
• Mitigation of Spatial Nonstationarity with Vision Transformers
• Efficient Deep Spiking Multi-Layer Perceptrons with Multiplication-Free Inference
• R-Mixup: Riemannian Mixup for Biological Networks
• SwinSF: Image Reconstruction from Spatial-Temporal Spike Streams