Dense Contrastive Learning for Self-Supervised Visual Pre-Training

Abstract

To date, most existing self-supervised learning methods are designed and optimized for image classification. These pre-trained models can be sub-optimal for dense prediction tasks due to the discrepancy between image-level prediction and pixel-level prediction. To fill this gap, we aim to design an effective, dense self-supervised learning method that directly works at the level of pixels (or local features) by taking into account the correspondence between local features. We present dense contrastive learning, which implements self-supervised learning by optimizing a pairwise contrastive (dis)similarity loss at the pixel level between two views of input images. Compared to the baseline method MoCo-v2, our method introduces negligible computation overhead (only <1% slower), but demonstrates consistently superior performance when transferring to downstream dense prediction tasks including object detection, semantic segmentation and instance segmentation; and outperforms the state-of-the-art methods by a large margin. Specifically, over the strong MoCo-v2 baseline, our method achieves significant improvements of 2.0% AP on PASCAL VOC object detection, 1.1% AP on COCO object detection, 0.9% AP on COCO instance segmentation, 3.0% mIoU on PASCAL VOC semantic segmentation and 1.8% mIoU on Cityscapes semantic segmentation. Code is available at: https://git.io/AdelaiDet

11 pages. Accepted to IEEE/CVF Conf. Comp. Vision Pattern Recognition (CVPR) 2021; Oral paper

References in corpus (9)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Baselines with Momentum Contrastive Learning
• Contrastive Multiview Coding
• Self-supervised Co-training for Video Representation Learning
• Universal Correspondence Network
• PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding
• An Analysis of Pre-Training on Object Detection
• Cheaper Pre-training Lunch: An Efficient Paradigm for Object Detection

Cited by in corpus (7)

• Aligning Pretraining for Detection via Object-Level Contrastive Learning
• Context-self contrastive pretraining for crop type semantic segmentation
• MST: Masked Self-Supervised Transformer for Visual Representation
• Semantic Distribution-aware Contrastive Adaptation for Semantic Segmentation
• Investigating the Role of Negatives in Contrastive Representation Learning
• Unsupervised Object-Level Representation Learning from Scene Images
• Dense Semantic Contrast for Self-Supervised Visual Representation Learning