MST: Masked Self-Supervised Transformer for Visual Representation

Abstract

Transformer has been widely used for self-supervised pre-training in Natural Language Processing (NLP) and achieved great success. However, it has not been fully explored in visual self-supervised learning. Meanwhile, previous methods only consider the high-level feature and learning representation from a global perspective, which may fail to transfer to the downstream dense prediction tasks focusing on local features. In this paper, we present a novel Masked Self-supervised Transformer approach named MST, which can explicitly capture the local context of an image while preserving the global semantic information. Specifically, inspired by the Masked Language Modeling (MLM) in NLP, we propose a masked token strategy based on the multi-head self-attention map, which dynamically masks some tokens of local patches without damaging the crucial structure for self-supervised learning. More importantly, the masked tokens together with the remaining tokens are further recovered by a global image decoder, which preserves the spatial information of the image and is more friendly to the downstream dense prediction tasks. The experiments on multiple datasets demonstrate the effectiveness and generality of the proposed method. For instance, MST achieves Top-1 accuracy of 76.9% with DeiT-S only using 300-epoch pre-training by linear evaluation, which outperforms supervised methods with the same epoch by 0.4% and its comparable variant DINO by 1.0\%. For dense prediction tasks, MST also achieves 42.7% mAP on MS COCO object detection and 74.04% mIoU on Cityscapes segmentation only with 100-epoch pre-training.

arXiv commentAccepted in NeurIPS 2021

References in corpus (12)

• Decoupled Weight Decay Regularization
• A Simple Framework for Contrastive Learning of Visual Representations
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Improved Baselines with Momentum Contrastive Learning
• Learning deep representations by mutual information estimation and maximization
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Momentum Contrast for Unsupervised Visual Representation Learning
• Self-Supervised Learning with Swin Transformers
• Self-EMD: Self-Supervised Object Detection without ImageNet
• Dense Contrastive Learning for Self-Supervised Visual Pre-Training
• ProTo: Program-Guided Transformer for Program-Guided Tasks
• Instance Localization for Self-supervised Detection Pretraining

Cited by in corpus (4)

• A Survey on Visual Transformer
• What to Hide from Your Students: Attention-Guided Masked Image Modeling
• A Survey of Visual Transformers
• Self-supervised 3D anatomy segmentation using self-distilled masked image transformer (SMIT)