What Should Not Be Contrastive in Contrastive Learning

Abstract

Recent self-supervised contrastive methods have been able to produce impressive transferable visual representations by learning to be invariant to different data augmentations. However, these methods implicitly assume a particular set of representational invariances (e.g., invariance to color), and can perform poorly when a downstream task violates this assumption (e.g., distinguishing red vs. yellow cars). We introduce a contrastive learning framework which does not require prior knowledge of specific, task-dependent invariances. Our model learns to capture varying and invariant factors for visual representations by constructing separate embedding spaces, each of which is invariant to all but one augmentation. We use a multi-head network with a shared backbone which captures information across each augmentation and alone outperforms all baselines on downstream tasks. We further find that the concatenation of the invariant and varying spaces performs best across all tasks we investigate, including coarse-grained, fine-grained, and few-shot downstream classification tasks, and various data corruptions.

arXiv commentPublished as a conference paper at ICLR 2021

References in corpus (4)

• A Simple Framework for Contrastive Learning of Visual Representations
• Improved Baselines with Momentum Contrastive Learning
• Contrastive Multiview Coding
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering

Cited by in corpus (8)

• Graph Contrastive Learning with Adaptive Augmentation
• Contrastive Representation Learning: A Framework and Review
• CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understanding
• Contrastive Self-supervised Sequential Recommendation with Robust Augmentation
• A Review of Predictive and Contrastive Self-supervised Learning for Medical Images
• Graph Communal Contrastive Learning
• Iterative Human and Automated Identification of Wildlife Images
• Intelligently Augmented Contrastive Tensor Factorization: Empowering Multi-dimensional Time Series Classification in Low-Data Environments