Addressing Feature Suppression in Unsupervised Visual Representations
arXiv:2012.09962
Abstract
Contrastive learning is one of the fastest growing research areas in machine learning due to its ability to learn useful representations without labeled data. However, contrastive learning is susceptible to feature suppression, i.e., it may discard important information relevant to the task of interest, and learn irrelevant features. Past work has addressed this limitation via handcrafted data augmentations that eliminate irrelevant information. This approach however does not work across all datasets and tasks. Further, data augmentations fail in addressing feature suppression in multi-attribute classification when one attribute can suppress features relevant to other attributes. In this paper, we analyze the objective function of contrastive learning and formally prove that it is vulnerable to feature suppression. We then present predictive contrastive learning (PCL), a framework for learning unsupervised representations that are robust to feature suppression. The key idea is to force the learned representation to predict the input, and hence prevent it from discarding important information. Extensive experiments verify that PCL is robust to feature suppression and outperforms state-of-the-art contrastive learning methods on a variety of datasets and tasks.
arXiv commentThe first two authors contributed equally to this paper
References in corpus (16)
- A Simple Framework for Contrastive Learning of Visual Representations
- Bootstrap your own latent: A new approach to self-supervised Learning
- Improved Baselines with Momentum Contrastive Learning
- Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- Learning deep representations by mutual information estimation and maximization
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- What Makes for Good Views for Contrastive Learning?
- Contrastive Multiview Coding
- Exploring Simple Siamese Representation Learning
- Variational Autoencoder for Deep Learning of Images, Labels and Captions
- On Mutual Information Maximization for Representation Learning
- FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age
- Memory-augmented Dense Predictive Coding for Video Representation Learning
- Intriguing Properties of Contrastive Losses
- Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learning
- Towards Fine-grained Visual Representations by Combining Contrastive Learning with Image Reconstruction and Attention-weighted Pooling