Contrastive Representations for Label Noise Require Fine-Tuning

Abstract

In this paper we show that the combination of a Contrastive representation with a label noise-robust classification head requires fine-tuning the representation in order to achieve state-of-the-art performances. Since fine-tuned representations are shown to outperform frozen ones, one can conclude that noise-robust classification heads are indeed able to promote meaningful representations if provided with a suitable starting point. Experiments are conducted to draw a comprehensive picture of performances by featuring six methods and nine noise instances of three different kinds (none, symmetric, and asymmetric). In presence of noise the experiments show that fine tuning of Contrastive representation allows the six methods to achieve better results than end-to-end learning and represent a new reference compare to the recent state of art. Results are also remarkable stable versus the noise level.

References in corpus (10)

• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Baselines with Momentum Contrastive Learning
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Normalizing Flows: An Introduction and Review of Current Methods
• Classification with Noisy Labels by Importance Reweighting
• Information-Theoretical Learning of Discriminative Clusters for Unsupervised Domain Adaptation
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• Meta Transition Adaptation for Robust Deep Learning with Noisy Labels
• Contrastive Learning Improves Model Robustness Under Label Noise
• Combining Self-Supervised and Supervised Learning with Noisy Labels