Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases

Abstract

Self-supervised representation learning approaches have recently surpassed their supervised learning counterparts on downstream tasks like object detection and image classification. Somewhat mysteriously the recent gains in performance come from training instance classification models, treating each image and it's augmented versions as samples of a single class. In this work, we first present quantitative experiments to demystify these gains. We demonstrate that approaches like MOCO and PIRL learn occlusion-invariant representations. However, they fail to capture viewpoint and category instance invariance which are crucial components for object recognition. Second, we demonstrate that these approaches obtain further gains from access to a clean object-centric training dataset like Imagenet. Finally, we propose an approach to leverage unstructured videos to learn representations that possess higher viewpoint invariance. Our results show that the learned representations outperform MOCOv2 trained on the same data in terms of invariances encoded and the performance on downstream image classification and semantic segmentation tasks.

References in corpus (1)

• Grad-CAM: Why did you say that?

Cited by in corpus (12)

• Contrastive Representation Learning: A Framework and Review
• Self-Damaging Contrastive Learning
• Move to See Better: Self-Improving Embodied Object Detection
• Towards Good Practices in Self-supervised Representation Learning
• Understanding Hyperbolic Metric Learning through Hard Negative Sampling
• Self-Supervised Ranking for Representation Learning
• Multi-Task Self-Training for Learning General Representations
• Cluster Analysis with Deep Embeddings and Contrastive Learning
• Watching Too Much Television is Good: Self-Supervised Audio-Visual Representation Learning from Movies and TV Shows
• On the robustness of self-supervised representations for multi-view object classification
• The Effects of Image Distribution and Task on Adversarial Robustness
• Contrastive learning of strong-mixing continuous-time stochastic processes