Revisiting Self-Supervised Visual Representation Learning

Abstract

Unsupervised visual representation learning remains a largely unsolved problem in computer vision research. Among a big body of recently proposed approaches for unsupervised learning of visual representations, a class of self-supervised techniques achieves superior performance on many challenging benchmarks. A large number of the pretext tasks for self-supervised learning have been studied, but other important aspects, such as the choice of convolutional neural networks (CNN), has not received equal attention. Therefore, we revisit numerous previously proposed self-supervised models, conduct a thorough large scale study and, as a result, uncover multiple crucial insights. We challenge a number of common practices in selfsupervised visual representation learning and observe that standard recipes for CNN design do not always translate to self-supervised representation learning. As part of our study, we drastically boost the performance of previously proposed techniques and outperform previously published state-of-the-art results by a large margin.

All three authors contributed equally. Code is available at https://github.com/google/revisiting-self-supervised

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Improving neural networks by preventing co-adaptation of feature detectors
• In Defense of the Triplet Loss for Person Re-Identification
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Invertible Residual Networks
• Improvements to context based self-supervised learning
• Time-Contrastive Networks: Self-Supervised Learning from Video
• Robustness via Retrying: Closed-Loop Robotic Manipulation with Self-Supervised Learning

Cited by in corpus (33)

• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Graph Contrastive Learning with Augmentations
• Learning Representations by Maximizing Mutual Information Across Views
• On Mutual Information Maximization for Representation Learning
• Audiovisual SlowFast Networks for Video Recognition
• Self-labelling via simultaneous clustering and representation learning
• Adversarial Self-Supervised Contrastive Learning
• What makes instance discrimination good for transfer learning?
• Selfie: Self-supervised Pretraining for Image Embedding
• A Framework For Contrastive Self-Supervised Learning And Designing A New Approach
• A Survey of Deep Learning for Scientific Discovery
• Unsupervised State Representation Learning in Atari
• Deep Comprehensive Correlation Mining for Image Clustering
• CLAR: Contrastive Learning of Auditory Representations
• Two Stream Networks for Self-Supervised Ego-Motion Estimation
• Improving Semantic Analysis on Point Clouds via Auxiliary Supervision of Local Geometric Priors
• How Useful is Self-Supervised Pretraining for Visual Tasks?
• CSNNs: Unsupervised, Backpropagation-free Convolutional Neural Networks for Representation Learning
• Online Deep Clustering for Unsupervised Representation Learning
• ClusterFit: Improving Generalization of Visual Representations
• Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
• Constrained Contrastive Distribution Learning for Unsupervised Anomaly Detection and Localisation in Medical Images
• Boosting Supervision with Self-Supervision for Few-shot Learning
• Towards a Hypothesis on Visual Transformation based Self-Supervision
• LPaintB: Learning to Paint from Self-Supervision
• Meta-SVDD: Probabilistic Meta-Learning for One-Class Classification in Cancer Histology Images
• StackMix: A complementary Mix algorithm
• AETv2: AutoEncoding Transformations for Self-Supervised Representation Learning by Minimizing Geodesic Distances in Lie Groups
• TVDIM: Enhancing Image Self-Supervised Pretraining via Noisy Text Data
• Towards Robust Pattern Recognition: A Review
• Reliable Label Bootstrapping for Semi-Supervised Learning
• Self-supervised classification of dynamic obstacles using the temporal information provided by videos
• Self-Supervised Visual Representation Learning Using Lightweight Architectures