Dense Semantic Contrast for Self-Supervised Visual Representation Learning

Abstract

Self-supervised representation learning for visual pre-training has achieved remarkable success with sample (instance or pixel) discrimination and semantics discovery of instance, whereas there still exists a non-negligible gap between pre-trained model and downstream dense prediction tasks. Concretely, these downstream tasks require more accurate representation, in other words, the pixels from the same object must belong to a shared semantic category, which is lacking in the previous methods. In this work, we present Dense Semantic Contrast (DSC) for modeling semantic category decision boundaries at a dense level to meet the requirement of these tasks. Furthermore, we propose a dense cross-image semantic contrastive learning framework for multi-granularity representation learning. Specially, we explicitly explore the semantic structure of the dataset by mining relations among pixels from different perspectives. For intra-image relation modeling, we discover pixel neighbors from multiple views. And for inter-image relations, we enforce pixel representation from the same semantic class to be more similar than the representation from different classes in one mini-batch. Experimental results show that our DSC model outperforms state-of-the-art methods when transferring to downstream dense prediction tasks, including object detection, semantic segmentation, and instance segmentation. Code will be made available.

arXiv commentACM MM 2021 Oral

References in corpus (17)

• Improved Baselines with Momentum Contrastive Learning
• Exploring Simple Siamese Representation Learning
• Unsupervised Deep Learning by Neighbourhood Discovery
• Unsupervised Learning of Dense Visual Representations
• Dense Contrastive Learning for Self-Supervised Visual Pre-Training
• Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning
• Two-Level Residual Distillation based Triple Network for Incremental Object Detection
• Exploring Instance Relations for Unsupervised Feature Embedding
• Deep Clustering by Semantic Contrastive Learning
• Video 3D Sampling for Self-supervised Representation Learning
• GATCluster: Self-Supervised Gaussian-Attention Network for Image Clustering
• Gaussian Constrained Attention Network for Scene Text Recognition
• Online Deep Clustering for Unsupervised Representation Learning
• FC2RN: A Fully Convolutional Corner Refinement Network for Accurate Multi-Oriented Scene Text Detection
• MMF: Multi-Task Multi-Structure Fusion for Hierarchical Image Classification
• Progressive Cluster Purification for Unsupervised Feature Learning
• Self-Training for Domain Adaptive Scene Text Detection