Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases

Abstract

Self-supervised representation learning approaches have recently surpassed their supervised learning counterparts on downstream tasks like object detection and image classification. Somewhat mysteriously the recent gains in performance come from training instance classification models, treating each image and it's augmented versions as samples of a single class. In this work, we first present quantitative experiments to demystify these gains. We demonstrate that approaches like MOCO and PIRL learn occlusion-invariant representations. However, they fail to capture viewpoint and category instance invariance which are crucial components for object recognition. Second, we demonstrate that these approaches obtain further gains from access to a clean object-centric training dataset like Imagenet. Finally, we propose an approach to leverage unstructured videos to learn representations that possess higher viewpoint invariance. Our results show that the learned representations outperform MOCOv2 trained on the same data in terms of invariances encoded and the performance on downstream image classification and semantic segmentation tasks.

References in corpus (9)

• Auto-Encoding Variational Bayes
• A Simple Framework for Contrastive Learning of Visual Representations
• Improved Baselines with Momentum Contrastive Learning
• GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild
• YFCC100M: The New Data in Multimedia Research
• Tutorial on Variational Autoencoders
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Contrastive Multiview Coding
• Grad-CAM: Why did you say that?

Cited by in corpus (31)

• Contrastive Representation Learning: A Framework and Review
• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• Video Transformers: A Survey
• Active, Continual Fine Tuning of Convolutional Neural Networks for Reducing Annotation Efforts
• Intriguing Properties of Contrastive Losses
• Contrastive Learning with Hard Negative Samples
• Understanding Self-supervised Learning with Dual Deep Networks
• Self-supervised Video Representation Learning by Uncovering Spatio-temporal Statistics
• Viewmaker Networks: Learning Views for Unsupervised Representation Learning
• A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning
• Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation
• Self-Damaging Contrastive Learning
• Move to See Better: Self-Improving Embodied Object Detection
• Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction
• SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
• Towards Good Practices in Self-supervised Representation Learning
• Spatially Consistent Representation Learning
• Supporting Clustering with Contrastive Learning
• Understanding Hyperbolic Metric Learning through Hard Negative Sampling
• Self-Supervised Ranking for Representation Learning
• Multi-Task Self-Training for Learning General Representations
• Cluster Analysis with Deep Embeddings and Contrastive Learning
• Watching Too Much Television is Good: Self-Supervised Audio-Visual Representation Learning from Movies and TV Shows
• On the robustness of self-supervised representations for multi-view object classification
• Object-Aware Cropping for Self-Supervised Learning
• The Effects of Image Distribution and Task on Adversarial Robustness
• Contrastive learning of strong-mixing continuous-time stochastic processes
• SERE: Exploring Feature Self-relation for Self-supervised Transformer
• Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency
• A Self-supervised Learning System for Object Detection in Videos Using Random Walks on Graphs
• DyStaB: Unsupervised Object Segmentation via Dynamic-Static Bootstrapping