Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels

Abstract

The success of learning with noisy labels (LNL) methods relies heavily on the success of a warm-up stage where standard supervised training is performed using the full (noisy) training set. In this paper, we identify a "warm-up obstacle": the inability of standard warm-up stages to train high quality feature extractors and avert memorization of noisy labels. We propose "Contrast to Divide" (C2D), a simple framework that solves this problem by pre-training the feature extractor in a self-supervised fashion. Using self-supervised pre-training boosts the performance of existing LNL approaches by drastically reducing the warm-up stage's susceptibility to noise level, shortening its duration, and improving extracted feature quality. C2D works out of the box with existing methods and demonstrates markedly improved performance, especially in the high noise regime, where we get a boost of more than 27% for CIFAR-100 with 90% noise over the previous state of the art. In real-life noise settings, C2D trained on mini-WebVision outperforms previous works both in WebVision and ImageNet validation sets by 3% top-1 accuracy. We perform an in-depth analysis of the framework, including investigating the performance of different pre-training approaches and estimating the effective upper bound of the LNL performance with semi-supervised learning. Code for reproducing our experiments is available at https://github.com/ContrastToDivide/C2D

References in corpus (34)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Decoupled Weight Decay Regularization
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Unsupervised Data Augmentation for Consistency Training
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Unsupervised Representation Learning by Predicting Image Rotations
• Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels
• Understanding deep learning requires rethinking generalization
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• What Makes for Good Views for Contrastive Learning?
• MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
• DivideMix: Learning with Noisy Labels as Semi-supervised Learning
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Exploring Simple Siamese Representation Learning
• A Closer Look at Memorization in Deep Networks
• Billion-scale semi-supervised learning for image classification
• WebVision Database: Visual Learning and Understanding from Web Data
• Early-Learning Regularization Prevents Memorization of Noisy Labels
• Self-training with Noisy Student improves ImageNet classification
• Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
• Are Anchor Points Really Indispensable in Label-Noise Learning?
• Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
• Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels
• Supervised Contrastive Learning
• Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learning
• Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels
• PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding
• Identifying Mislabeled Data using the Area Under the Margin Ranking
• SOSELETO: A Unified Approach to Transfer Learning and Training with Noisy Labels
• Robust and On-the-fly Dataset Denoising for Image Classification

Cited by in corpus (7)

• Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels
• SSR: An Efficient and Robust Framework for Learning with Unknown Label Noise
• Benchmarking Conventional Vision Models on Neuromorphic Fall Detection and Action Recognition Dataset
• Consistency Regularization Can Improve Robustness to Label Noise
• Instance-dependent Noisy-label Learning with Graphical Model Based Noise-rate Estimation
• CoDiM: Learning with Noisy Labels via Contrastive Semi-Supervised Learning
• Contrastive Representations for Label Noise Require Fine-Tuning