Self-labelling via simultaneous clustering and representation learning

Abstract

Combining clustering and representation learning is one of the most promising approaches for unsupervised learning of deep neural networks. However, doing so naively leads to ill posed learning problems with degenerate solutions. In this paper, we propose a novel and principled learning formulation that addresses these issues. The method is obtained by maximizing the information between labels and input data indices. We show that this criterion extends standard crossentropy minimization to an optimal transport problem, which we solve efficiently for millions of input images and thousands of labels using a fast variant of the Sinkhorn-Knopp algorithm. The resulting method is able to self-label visual data so as to train highly competitive image representations without manual labels. Our method achieves state of the art representation learning performance for AlexNet and ResNet-50 on SVHN, CIFAR-10, CIFAR-100 and ImageNet and yields the first self-supervised AlexNet that outperforms the supervised Pascal VOC detection baseline. Code and models are available.

Accepted paper at the International Conference on Learning Representations (ICLR) 2020

References in corpus (8)

• Unsupervised Learning of Video Representations using LSTMs
• One weird trick for parallelizing convolutional neural networks
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Learning Representations by Maximizing Mutual Information Across Views
• Improvements to context based self-supervised learning
• A critical analysis of self-supervision, or what we can learn from a single image
• Unsupervised Deep Learning by Neighbourhood Discovery
• Revisiting Self-Supervised Visual Representation Learning

Cited by in corpus (26)

• Twin Contrastive Learning for Online Clustering
• One Million Scenes for Autonomous Driving: ONCE Dataset
• Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
• Large-scale Unsupervised Semantic Segmentation
• Deep Robust Clustering by Contrastive Learning
• Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation
• Contrastive Learning with Stronger Augmentations
• Conditional Negative Sampling for Contrastive Learning of Visual Representations
• A Prototype-Oriented Framework for Unsupervised Domain Adaptation
• Scalable Optimal Transport Methods in Machine Learning: A Contemporary Survey
• AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries
• Data clustering: a fundamental method in data science and management
• Spatial and Temporal Hierarchy for Autonomous Navigation using Active Inference in Minigrid Environment
• A Generic Self-Supervised Framework of Learning Invariant Discriminative Features
• Self-Supervised Learning by Estimating Twin Class Distributions
• What is Wrong with One-Class Anomaly Detection?
• A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning
• Deep Contrastive Graph Representation via Adaptive Homotopy Learning
• Clustering-Based Representation Learning through Output Translation and Its Application to Remote--Sensing Images
• Loss Function Entropy Regularization for Diverse Decision Boundaries
• Double cycle-consistent generative adversarial network for unsupervised conditional generation
• Progressive Stage-wise Learning for Unsupervised Feature Representation Enhancement
• UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning
• Unsupervised Object Detection with LiDAR Clues
• Improving Image Clustering through Sample Ranking and Its Application to remote--sensing images
• Self-Supervised Learning for Large-Scale Unsupervised Image Clustering