Swin Transformer V2: Scaling Up Capacity and Resolution

Abstract

Large-scale NLP models have been shown to significantly improve the performance on language tasks with no signs of saturation. They also demonstrate amazing few-shot capabilities like that of human beings. This paper aims to explore large-scale models in computer vision. We tackle three major issues in training and application of large vision models, including training instability, resolution gaps between pre-training and fine-tuning, and hunger on labelled data. Three main techniques are proposed: 1) a residual-post-norm method combined with cosine attention to improve training stability; 2) A log-spaced continuous position bias method to effectively transfer models pre-trained using low-resolution images to downstream tasks with high-resolution inputs; 3) A self-supervised pre-training method, SimMIM, to reduce the needs of vast labeled images. Through these techniques, this paper successfully trained a 3 billion-parameter Swin Transformer V2 model, which is the largest dense vision model to date, and makes it capable of training with images of up to 1,536$\times$1,536 resolution. It set new performance records on 4 representative vision tasks, including ImageNet-V2 image classification, COCO object detection, ADE20K semantic segmentation, and Kinetics-400 video action classification. Also note our training is much more efficient than that in Google's billion-level visual models, which consumes 40 times less labelled data and 40 times less training time. Code is available at \url{https://github.com/microsoft/Swin-Transformer}.

References in corpus (23)

• Learning Transferable Visual Models From Natural Language Supervision
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Language Models are Few-Shot Learners
• The Kinetics Human Action Video Dataset
• Scaling Laws for Neural Language Models
• BEiT: BERT Pre-Training of Image Transformers
• CoAtNet: Marrying Convolution and Attention for All Data Sizes
• Training Deep Nets with Sublinear Memory Cost
• Do ImageNet Classifiers Generalize to ImageNet?
• You Only Learn One Representation: Unified Network for Multiple Tasks
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Focal Self-attention for Local-Global Interactions in Vision Transformers
• UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
• Self-supervised Pretraining of Visual Features in the Wild
• CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
• Rethinking Positional Encoding in Language Pre-training
• Dynamic Head: Unifying Object Detection Heads with Attentions
• Video Swin Transformer
• Scaling Vision with Sparse Mixture of Experts
• VOLO: Vision Outlooker for Visual Recognition
• Characterizing signal propagation to close the performance gap in unnormalized ResNets
• SimMIM: A Simple Framework for Masked Image Modeling
• TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?

Cited by in corpus (1)

• SimMIM: A Simple Framework for Masked Image Modeling