Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning

Abstract

Contrastive self-supervised learning (CSL) has attracted increasing attention for model pre-training via unlabeled data. The resulted CSL models provide instance-discriminative visual features that are uniformly scattered in the feature space. During deployment, the common practice is to directly fine-tune CSL models with cross-entropy, which however may not be the best strategy in practice. Although cross-entropy tends to separate inter-class features, the resulting models still have limited capability for reducing intra-class feature scattering that exists in CSL models. In this paper, we investigate whether applying contrastive learning to fine-tuning would bring further benefits, and analytically find that optimizing the contrastive loss benefits both discriminative representation learning and model optimization during fine-tuning. Inspired by these findings, we propose Contrast-regularized tuning (Core-tuning), a new approach for fine-tuning CSL models. Instead of simply adding the contrastive loss to the objective of fine-tuning, Core-tuning further applies a novel hard pair mining strategy for more effective contrastive fine-tuning, as well as smoothing the decision boundary to better exploit the learned discriminative feature space. Extensive experiments on image classification and semantic segmentation verify the effectiveness of Core-tuning.

NeurIPS 2021. Source code: https://github.com/Vanint/Core-tuning

References in corpus (10)

• Rethinking Atrous Convolution for Semantic Image Segmentation
• Theoretically Principled Trade-off between Robustness and Accuracy
• Collaborative Unsupervised Domain Adaptation for Medical Image Diagnosis
• Deep Long-Tailed Learning: A Survey
• COVID-DA: Deep Domain Adaptation from Typical Pneumonia to COVID-19
• MixCo: Mix-up Contrastive Learning for Visual Representation
• Conditional Negative Sampling for Contrastive Learning of Visual Representations
• Understanding the Behaviour of Contrastive Loss
• Bi-tuning of Pre-trained Representations
• RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayEr

Cited by in corpus (4)

• CrossCBR: Cross-view Contrastive Learning for Bundle Recommendation
• An Asymmetric Contrastive Loss for Handling Imbalanced Datasets
• Source-free Domain Adaptation via Avatar Prototype Generation and Adaptation
• Efficient Self-Supervision using Patch-based Contrastive Learning for Histopathology Image Segmentation