A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning

Abstract

For an image query, unsupervised contrastive learning labels crops of the same image as positives, and other image crops as negatives. Although intuitive, such a native label assignment strategy cannot reveal the underlying semantic similarity between a query and its positives and negatives, and impairs performance, since some negatives are semantically similar to the query or even share the same semantic class as the query. In this work, we first prove that for contrastive learning, inaccurate label assignment heavily impairs its generalization for semantic instance discrimination, while accurate labels benefit its generalization. Inspired by this theory, we propose a novel self-labeling refinement approach for contrastive learning. It improves the label quality via two complementary modules: (i) self-labeling refinery (SLR) to generate accurate labels and (ii) momentum mixup (MM) to enhance similarity between query and its positive. SLR uses a positive of a query to estimate semantic similarity between a query and its positive and negatives, and combines estimated similarity with vanilla label assignment in contrastive learning to iteratively generate more accurate and informative soft labels. We theoretically show that our SLR can exactly recover the true semantic labels of label-corrupted data, and supervises networks to achieve zero prediction error on classification tasks. MM randomly combines queries and positives to increase semantic similarity between the generated virtual queries and their positives so as to improves label accuracy. Experimental results on CIFAR10, ImageNet, VOC and COCO show the effectiveness of our method. PyTorch code and model will be released online.

arXiv commentunder review. arXiv admin note: substantial text overlap with arXiv:1903.11680 by other authors

References in corpus (22)

• Distilling the Knowledge in a Neural Network
• mixup: Beyond Empirical Risk Minimization
• Improved Baselines with Momentum Contrastive Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• When Does Label Smoothing Help?
• What Makes for Good Views for Contrastive Learning?
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Contrastive Multiview Coding
• Exploring Simple Siamese Representation Learning
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Empirical Bernstein Bounds and Sample Variance Penalization
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• Label Refinery: Improving ImageNet Classification through Label Progression
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Self-labelling via simultaneous clustering and representation learning
• Diverse Neural Network Learns True Target Functions
• i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning
• MixCo: Mix-up Contrastive Learning for Visual Representation
• Contrastive Learning with Stronger Augmentations
• CO2: Consistent Contrast for Unsupervised Visual Representation Learning
• Towards Domain-Agnostic Contrastive Learning