Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer

Abstract

Very recently, Window-based Transformers, which computed self-attention within non-overlapping local windows, demonstrated promising results on image classification, semantic segmentation, and object detection. However, less study has been devoted to the cross-window connection which is the key element to improve the representation ability. In this work, we revisit the spatial shuffle as an efficient way to build connections among windows. As a result, we propose a new vision transformer, named Shuffle Transformer, which is highly efficient and easy to implement by modifying two lines of code. Furthermore, the depth-wise convolution is introduced to complement the spatial shuffle for enhancing neighbor-window connections. The proposed architectures achieve excellent performance on a wide range of visual tasks including image-level classification, object detection, and semantic segmentation. Code will be released for reproduction.

References in corpus (13)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Longformer: The Long-Document Transformer
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Transformer in Transformer
• Generating Long Sequences with Sparse Transformers
• Escaping the Big Data Paradigm with Compact Transformers
• LocalViT: Analyzing Locality in Vision Transformers
• Stand-Alone Self-Attention in Vision Models
• CvT: Introducing Convolutions to Vision Transformers
• Sparse R-CNN: End-to-End Object Detection with Learnable Proposals
• Interlaced Sparse Self-Attention for Semantic Segmentation

Cited by in corpus (15)

• A Survey on Visual Transformer
• Transformers in Vision: A Survey
• Vision Transformers for Single Image Dehazing
• Point Cloud Classification Using Content-based Transformer via Clustering in Feature Space
• CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation
• Uformer: A General U-Shaped Transformer for Image Restoration
• Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets
• S${}^2$-MLPv2: Improved Spatial-Shift MLP Architecture for Vision
• A Convolutional Vision Transformer for Semantic Segmentation of Side-Scan Sonar Data
• ConvMLP: Hierarchical Convolutional MLPs for Vision
• UniNeXt: Exploring A Unified Architecture for Vision Recognition
• MSG-Transformer: Exchanging Local Spatial Information by Manipulating Messenger Tokens
• Rethinking Token-Mixing MLP for MLP-based Vision Backbone
• Shuffle Transformer with Feature Alignment for Video Face Parsing
• SwinSF: Image Reconstruction from Spatial-Temporal Spike Streams