Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Abstract

This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with \textbf{S}hifted \textbf{win}dows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures. The code and models are publicly available at~\url{https://github.com/microsoft/Swin-Transformer}.

Cited by in corpus (50)

• A Survey on Visual Transformer
• Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting
• A survey of the Vision Transformers and their CNN-Transformer based Variants
• Early Convolutions Help Transformers See Better
• Intriguing Properties of Vision Transformers
• Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity
• Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks
• SwinTrack: A Simple and Strong Baseline for Transformer Tracking
• ViDT: An Efficient and Effective Fully Transformer-based Object Detector
• A Survey of Visual Transformers
• DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting
• MetaFormer Is Actually What You Need for Vision
• Neuromorphic Perception and Navigation for Mobile Robots: A Review
• Infrared Small-Dim Target Detection with Transformer under Complex Backgrounds
• Deep Learning for Pancreas Segmentation: a Systematic Review
• An Empirical Study of Training End-to-End Vision-and-Language Transformers
• Multi-View Stereo with Transformer
• CovidExpert: A Triplet Siamese Neural Network framework for the detection of COVID-19
• MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation
• Pruning Self-attentions into Convolutional Layers in Single Path
• Self-distillation with Batch Knowledge Ensembling Improves ImageNet Classification
• AdaViT: Adaptive Vision Transformers for Efficient Image Recognition
• PU-Transformer: Point Cloud Upsampling Transformer
• PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer
• Oriented RepPoints for Aerial Object Detection
• MSG-Transformer: Exchanging Local Spatial Information by Manipulating Messenger Tokens
• A Comprehensive Study on Medical Image Segmentation using Deep Neural Networks
• Exploiting Both Domain-specific and Invariant Knowledge via a Win-win Transformer for Unsupervised Domain Adaptation
• AdaDM: Enabling Normalization for Image Super-Resolution
• M2MRF: Many-to-Many Reassembly of Features for Tiny Lesion Segmentation in Fundus Images
• An Image Patch is a Wave: Phase-Aware Vision MLP
• A Unified Pruning Framework for Vision Transformers
• Toward Accurate and Reliable Iris Segmentation Using Uncertainty Learning
• Pyramid Adversarial Training Improves ViT Performance
• Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation
• Learning to Transfer for Traffic Forecasting via Multi-task Learning
• OAH-Net: A Deep Neural Network for Hologram Reconstruction of Off-axis Digital Holographic Microscope
• Bridging Global Context Interactions for High-Fidelity Image Completion
• DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers
• Evaluating Transformers for Lightweight Action Recognition
• PatchFormer: An Efficient Point Transformer with Patch Attention
• Video Frame Interpolation Transformer
• TransMix: Attend to Mix for Vision Transformers
• Trans4Trans: Efficient Transformer for Transparent Object Segmentation to Help Visually Impaired People Navigate in the Real World
• RaftMLP: How Much Can Be Done Without Attention and with Less Spatial Locality?
• PointMixer: MLP-Mixer for Point Cloud Understanding
• Open-World Entity Segmentation
• A Lightweight Graph Transformer Network for Human Mesh Reconstruction from 2D Human Pose
• edge-SR: Super-Resolution For The Masses
• Towards Comprehensive Monocular Depth Estimation: Multiple Heads Are Better Than One