What Makes for Good Views for Contrastive Learning?

Abstract

Contrastive learning between multiple views of the data has recently achieved state of the art performance in the field of self-supervised representation learning. Despite its success, the influence of different view choices has been less studied. In this paper, we use theoretical and empirical analysis to better understand the importance of view selection, and argue that we should reduce the mutual information (MI) between views while keeping task-relevant information intact. To verify this hypothesis, we devise unsupervised and semi-supervised frameworks that learn effective views by aiming to reduce their MI. We also consider data augmentation as a way to reduce MI, and show that increasing data augmentation indeed leads to decreasing MI and improves downstream classification accuracy. As a by-product, we achieve a new state-of-the-art accuracy on unsupervised pre-training for ImageNet classification ($73\%$ top-1 linear readout with a ResNet-50). In addition, transferring our models to PASCAL VOC object detection and COCO instance segmentation consistently outperforms supervised pre-training. Code:http://github.com/HobbitLong/PyContrast

NeurIPS 2020. Project page: https://hobbitlong.github.io/InfoMin/

References in corpus (32)

• Adam: A Method for Stochastic Optimization
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distributed Representations of Words and Phrases and their Compositionality
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• The information bottleneck method
• Improved Baselines with Momentum Contrastive Learning
• Learning deep representations by mutual information estimation and maximization
• Unsupervised Learning of Video Representations using LSTMs
• NICE: Non-linear Independent Components Estimation
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Density estimation using Real NVP
• Learning Representations by Maximizing Mutual Information Across Views
• Contrastive Multiview Coding
• MINE: Mutual Information Neural Estimation
• An efficient framework for learning sentence representations
• Large Scale Adversarial Representation Learning
• Deep Variational Information Bottleneck
• On Mutual Information Maximization for Representation Learning
• Learning Video Representations using Contrastive Bidirectional Transformer
• Perfect match: Improved cross-modal embeddings for audio-visual synchronisation
• A critical analysis of self-supervision, or what we can learn from a single image
• The Conditional Entropy Bottleneck
• A Mutual Information Maximization Perspective of Language Representation Learning
• Local Aggregation for Unsupervised Learning of Visual Embeddings
• Learning visual groups from co-occurrences in space and time
• Formal Limitations on the Measurement of Mutual Information
• Automatic Shortcut Removal for Self-Supervised Representation Learning
• Watching the World Go By: Representation Learning from Unlabeled Videos
• Learning Spatiotemporal Features via Video and Text Pair Discrimination
• Unsupervised Learning from Video with Deep Neural Embeddings

Cited by in corpus (135)

• Bootstrap your own latent: A new approach to self-supervised Learning
• On the Opportunities and Risks of Foundation Models
• Graph Contrastive Learning with Adaptive Augmentation
• Contrastive Representation Learning: A Framework and Review
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances
• Hard Negative Mixing for Contrastive Learning
• Self-Supervised MultiModal Versatile Networks
• Automated Self-Supervised Learning for Recommendation
• Semantics-Guided Contrastive Network for Zero-Shot Object detection
• Supervised Contrastive Learning
• Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
• Self-Supervised Multisensor Change Detection
• COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?
• HyperNet: Self-Supervised Hyperspectral Spatial-Spectral Feature Understanding Network for Hyperspectral Change Detection
• Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data
• Adversarial Self-Supervised Contrastive Learning
• Efficient Self-supervised Vision Transformers for Representation Learning
• Spatiotemporal Contrastive Video Representation Learning
• Self-Supervised Representation Learning for Astronomical Images
• Automated Spatio-Temporal Graph Contrastive Learning
• Partial success in closing the gap between human and machine vision
• Intriguing Properties of Contrastive Losses
• InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective
• Are all negatives created equal in contrastive instance discrimination?
• Aligning Pretraining for Detection via Object-Level Contrastive Learning
• i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning
• A Generic Fundus Image Enhancement Network Boosted by Frequency Self-supervised Representation Learning
• On Mutual Information in Contrastive Learning for Visual Representations
• Unsupervised Contrastive Learning based Transformer for Lung Nodule Detection
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
• Representation Learning via Invariant Causal Mechanisms
• LogME: Practical Assessment of Pre-trained Models for Transfer Learning
• Multi-Label Contrastive Learning for Abstract Visual Reasoning
• Contrastive Learning with Hard Negative Samples
• Contrastive Learning with Stronger Augmentations
• Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations
• Cross-Modal Contrastive Learning for Text-to-Image Generation
• VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning
• Do Different Tracking Tasks Require Different Appearance Models?
• Cross-Camera Feature Prediction for Intra-Camera Supervised Person Re-identification across Distant Scenes
• Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning
• Contrastive Learning Inverts the Data Generating Process
• MedAug: Contrastive learning leveraging patient metadata improves representations for chest X-ray interpretation
• Conditional Negative Sampling for Contrastive Learning of Visual Representations
• Detecting Arbitrary Order Beneficial Feature Interactions for Recommender Systems
• Understanding the Behaviour of Contrastive Loss
• Improving Text-to-Image Synthesis Using Contrastive Learning
• Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning
• Revisiting Locally Supervised Learning: an Alternative to End-to-end Training
• Viewmaker Networks: Learning Views for Unsupervised Representation Learning
• Surrogate- and invariance-boosted contrastive learning for data-scarce applications in science
• Towards the Generalization of Contrastive Self-Supervised Learning
• On the surprising similarities between supervised and self-supervised models
• Self-Knowledge Distillation based Self-Supervised Learning for Covid-19 Detection from Chest X-Ray Images
• Multi-queue Momentum Contrast for Microvideo-Product Retrieval
• Self-supervised Pre-training with Hard Examples Improves Visual Representations
• Universum-inspired Supervised Contrastive Learning
• Supervised Contrastive Learning with Hard Negative Samples
• Boosting Discriminative Visual Representation Learning with Scenario-Agnostic Mixup
• Open-Set Hypothesis Transfer with Semantic Consistency
• Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination
• Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction
• Group Identification via Transitional Hypergraph Convolution with Cross-view Self-supervised Learning
• Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubs
• Self-supervised Learning from a Multi-view Perspective
• Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity
• Iterative Graph Self-Distillation
• Temperature as Uncertainty in Contrastive Learning
• Understand and Improve Contrastive Learning Methods for Visual Representation: A Review
• Spatially Consistent Representation Learning
• Improving Transformation Invariance in Contrastive Representation Learning
• Improving Transferability of Representations via Augmentation-Aware Self-Supervision
• Self-Supervised Learning for Gastritis Detection with Gastric X-ray Images
• Evolution Is All You Need: Phylogenetic Augmentation for Contrastive Learning
• Instance Localization for Self-supervised Detection Pretraining
• Unsupervised Object-Level Representation Learning from Scene Images
• For self-supervised learning, Rationality implies generalization, provably
• Augmentation-aware Self-supervised Learning with Conditioned Projector
• Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning
• Generative Models as a Data Source for Multiview Representation Learning
• Calibrating and Improving Graph Contrastive Learning
• Representation Learning via Consistent Assignment of Views to Clusters
• Self-Supervised Learning by Estimating Twin Class Distributions
• DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation
• Evaluating the Label Efficiency of Contrastive Self-Supervised Learning for Multi-Resolution Satellite Imagery
• A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning
• Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning
• Bootstrap Equilibrium and Probabilistic Speaker Representation Learning for Self-supervised Speaker Verification
• Self-supervised pre-training and contrastive representation learning for multiple-choice video QA
• On Feature Decorrelation in Self-Supervised Learning
• Distribution Estimation to Automate Transformation Policies for Self-Supervision
• MV-MR: multi-views and multi-representations for self-supervised learning and knowledge distillation
• An Empirical Study and Analysis on Open-Set Semi-Supervised Learning
• Improving Contrastive Learning by Visualizing Feature Transformation
• Center-wise Local Image Mixture For Contrastive Representation Learning
• EqCo: Equivalent Rules for Self-supervised Contrastive Learning
• Function Contrastive Learning of Transferable Meta-Representations
• Heterogeneous Contrastive Learning: Encoding Spatial Information for Compact Visual Representations
• Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency
• Contrastive Attraction and Contrastive Repulsion for Representation Learning
• Trapped in texture bias? A large scale comparison of deep instance segmentation
• Compressive Visual Representations
• Self-Supervised Pretraining on Satellite Imagery: a Case Study on Label-Efficient Vehicle Detection
• Addressing Feature Suppression in Unsupervised Visual Representations
• Contrastive Object-level Pre-training with Spatial Noise Curriculum Learning
• Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations
• Robust Representation Learning via Perceptual Similarity Metrics
• A Unified Mixture-View Framework for Unsupervised Representation Learning
• Contrastive Learning of Global-Local Video Representations
• Understanding and Achieving Efficient Robustness with Adversarial Supervised Contrastive Learning
• Contrast-reconstruction Representation Learning for Self-supervised Skeleton-based Action Recognition
• Feature Distillation With Guided Adversarial Contrastive Learning
• Identical and Fraternal Twins: Fine-Grained Semantic Contrastive Learning of Sentence Representations
• Complementary Relation Contrastive Distillation
• Exploring Set Similarity for Dense Self-supervised Representation Learning
• ParamCrop: Parametric Cubic Cropping for Video Contrastive Learning
• Object-Aware Cropping for Self-Supervised Learning
• K-Shot Contrastive Learning of Visual Features with Multiple Instance Augmentations
• Contrastive Learning with Temporal Correlated Medical Images: A Case Study using Lung Segmentation in Chest X-Rays
• Equivariant Contrastive Learning
• Self-Supervised Learning for Large-Scale Unsupervised Image Clustering
• OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learning
• On Compositions of Transformations in Contrastive Self-Supervised Learning
• Contrastive Representation Learning with Trainable Augmentation Channel
• Learning to See by Looking at Noise
• Recent Advancements in Self-Supervised Paradigms for Visual Feature Representation
• Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency
• Characterizing and Improving the Robustness of Self-Supervised Learning through Background Augmentations
• Contrastive learning of strong-mixing continuous-time stochastic processes
• Pretext Tasks selection for multitask self-supervised speech representation learning
• Unsupervised Representation Learning for Binary Networks by Joint Classifier Learning