Whitening for Self-Supervised Representation Learning

Abstract

Most of the current self-supervised representation learning (SSL) methods are based on the contrastive loss and the instance-discrimination task, where augmented versions of the same image instance ("positives") are contrasted with instances extracted from other images ("negatives"). For the learning to be effective, many negatives should be compared with a positive pair, which is computationally demanding. In this paper, we propose a different direction and a new loss function for SSL, which is based on the whitening of the latent-space features. The whitening operation has a "scattering" effect on the batch samples, avoiding degenerate solutions where all the sample representations collapse to a single point. Our solution does not require asymmetric networks and it is conceptually simple. Moreover, since negatives are not needed, we can extract multiple positive pairs from the same image instance. The source code of the method and of all the experiments is available at: https://github.com/htdt/self-supervised.

ICML 2021

References in corpus (27)

• Adam: A Method for Stochastic Optimization
• Efficient Estimation of Word Representations in Vector Space
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distributed Representations of Words and Phrases and their Compositionality
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Bootstrap your own latent: A new approach to self-supervised Learning
• In Defense of the Triplet Loss for Person Re-Identification
• Improved Regularization of Convolutional Neural Networks with Cutout
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Learning deep representations by mutual information estimation and maximization
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Large Batch Training of Convolutional Networks
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Exploring Simple Siamese Representation Learning
• Large Scale Adversarial Representation Learning
• On Mutual Information Maximization for Representation Learning
• Stacked What-Where Auto-encoders
• Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
• Local Aggregation for Unsupervised Learning of Visual Embeddings
• Convolutional Clustering for Unsupervised Learning
• Decorrelated Batch Normalization
• Understanding Self-supervised Learning with Dual Deep Networks
• BYOL works even without batch statistics
• Whitening and Coloring batch transform for GANs

Cited by in corpus (10)

• Contrastive Representation Learning: A Framework and Review
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Neighborhood-Enhanced Supervised Contrastive Learning for Collaborative Filtering
• From Canonical Correlation Analysis to Self-supervised Graph Neural Networks
• Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction
• On Feature Decorrelation in Self-Supervised Learning
• EqCo: Equivalent Rules for Self-supervised Contrastive Learning
• Learnable Adaptive Cosine Estimator (LACE) for Image Classification
• Self-Supervised Learning for Large-Scale Unsupervised Image Clustering
• HoughCL: Finding Better Positive Pairs in Dense Self-supervised Learning