Self-Supervised Learning with Swin Transformers

Abstract

We are witnessing a modeling shift from CNN to Transformers in computer vision. In this work, we present a self-supervised learning approach called MoBY, with Vision Transformers as its backbone architecture. The approach basically has no new inventions, which is combined from MoCo v2 and BYOL and tuned to achieve reasonably high accuracy on ImageNet-1K linear evaluation: 72.8% and 75.0% top-1 accuracy using DeiT-S and Swin-T, respectively, by 300-epoch training. The performance is slightly better than recent works of MoCo v3 and DINO which adopt DeiT as the backbone, but with much lighter tricks. More importantly, the general-purpose Swin Transformer backbone enables us to also evaluate the learnt representations on downstream tasks such as object detection and semantic segmentation, in contrast to a few recent approaches built on ViT/DeiT which only report linear evaluation results on ImageNet-1K due to ViT/DeiT not tamed for these dense prediction tasks. We hope our results can facilitate more comprehensive evaluation of self-supervised learning methods designed for Transformer architectures. Our code and models are available at https://github.com/SwinTransformer/Transformer-SSL, which will be continually enriched.

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Decoupled Weight Decay Regularization
• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Baselines with Momentum Contrastive Learning

Cited by in corpus (15)

• A Survey on Visual Transformer
• Transformers in Vision: A Survey
• BEiT: BERT Pre-Training of Image Transformers
• XCiT: Cross-Covariance Image Transformers
• Localizing Objects with Self-Supervised Transformers and no Labels
• Efficient Self-supervised Vision Transformers for Representation Learning
• Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling
• Self-Supervised Pre-Training for Transformer-Based Person Re-Identification
• A Survey of Visual Transformers
• DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting
• SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Driving
• MST: Masked Self-Supervised Transformer for Visual Representation
• FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Models
• Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning
• SERE: Exploring Feature Self-relation for Self-supervised Transformer