Learning Representations by Maximizing Mutual Information Across Views

Abstract

We propose an approach to self-supervised representation learning based on maximizing mutual information between features extracted from multiple views of a shared context. For example, one could produce multiple views of a local spatio-temporal context by observing it from different locations (e.g., camera positions within a scene), and via different modalities (e.g., tactile, auditory, or visual). Or, an ImageNet image could provide a context from which one produces multiple views by repeatedly applying data augmentation. Maximizing mutual information between features extracted from these views requires capturing information about high-level factors whose influence spans multiple views -- e.g., presence of certain objects or occurrence of certain events. Following our proposed approach, we develop a model which learns image representations that significantly outperform prior methods on the tasks we consider. Most notably, using self-supervised learning, our model learns representations which achieve 68.1% accuracy on ImageNet using standard linear evaluation. This beats prior results by over 12% and concurrent results by 7%. When we extend our model to use mixture-based representations, segmentation behaviour emerges as a natural side-effect. Our code is available online: https://github.com/Philip-Bachman/amdim-public.

References in corpus (14)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distributed Representations of Words and Phrases and their Compositionality
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Learning deep representations by mutual information estimation and maximization
• Training Very Deep Networks
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• Contrastive Multiview Coding
• Reinforcement Learning with Deep Energy-Based Policies
• On Variational Bounds of Mutual Information
• Rethinking ImageNet Pre-training
• Revisiting Self-Supervised Visual Representation Learning
• Formal Limitations on the Measurement of Mutual Information
• Time-Contrastive Networks: Self-Supervised Learning from Video

Cited by in corpus (108)

• Improved Baselines with Momentum Contrastive Learning
• Graph Contrastive Learning with Adaptive Augmentation
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Contrastive Representation Learning: A Framework and Review
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Prototypical Contrastive Learning of Unsupervised Representations
• A Review on Deep Learning in UAV Remote Sensing
• Exploring Simple Siamese Representation Learning
• Image Quality Assessment using Contrastive Learning
• Hard Negative Mixing for Contrastive Learning
• A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives
• A survey on Semi-, Self- and Unsupervised Learning for Image Classification
• Momentum Contrastive Learning for Few-Shot COVID-19 Diagnosis from Chest CT Images
• TCLR: Temporal Contrastive Learning for Video Representation
• Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases
• SelfHAR: Improving Human Activity Recognition through Self-training with Unlabeled Data
• Social Recommendation with Self-Supervised Metagraph Informax Network
• Consistent Multiple Graph Embedding for Multi-View Clustering
• Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
• COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning
• Effectiveness of self-supervised pre-training for speech recognition
• Self-labelling via simultaneous clustering and representation learning
• Self-Contrastive Learning with Hard Negative Sampling for Self-supervised Point Cloud Learning
• A Mutual Information Maximization Perspective of Language Representation Learning
• Robust Pre-Training by Adversarial Contrastive Learning
• A Framework For Contrastive Self-Supervised Learning And Designing A New Approach
• Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift
• Cluster Contrast for Unsupervised Person Re-Identification
• A Survey of Deep Learning for Scientific Discovery
• A Review of Predictive and Contrastive Self-supervised Learning for Medical Images
• Jointly Contrastive Representation Learning on Road Network and Trajectory
• A Generic Fundus Image Enhancement Network Boosted by Frequency Self-supervised Representation Learning
• Unsupervised Deep Representation Learning and Few-Shot Classification of PolSAR Images
• Multi-Label Contrastive Learning for Abstract Visual Reasoning
• Contrastive Learning with Stronger Augmentations
• Self-supervised Contrastive Learning of Multi-view Facial Expressions
• PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers
• Pruning Convolutional Neural Networks with Self-Supervision
• CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients
• Contrastive Learning Inverts the Data Generating Process
• Contrastive Multi-Level Graph Neural Networks for Session-based Recommendation
• False: False Negative Samples Aware Contrastive Learning for Semantic Segmentation of High-Resolution Remote Sensing Image
• CoLES: Contrastive Learning for Event Sequences with Self-Supervision
• Unsupervised State Representation Learning in Atari
• Self-Supervised Learning For Few-Shot Image Classification
• Understanding the Behaviour of Contrastive Loss
• Evaluating Contrastive Models for Instance-based Image Retrieval
• Neural Transformation Learning for Deep Anomaly Detection Beyond Images
• Poisoning and Backdooring Contrastive Learning
• Unsupervised Two-Stage Anomaly Detection
• Contrastive encoder pre-training-based clustered federated learning for heterogeneous data
• When Does Self-supervision Improve Few-shot Learning?
• Modeling Multiple Views via Implicitly Preserving Global Consistency and Local Complementarity
• Global-Local Bidirectional Reasoning for Unsupervised Representation Learning of 3D Point Clouds
• Exploring Masked Autoencoders for Sensor-Agnostic Image Retrieval in Remote Sensing
• Unsupervised Deep Metric Learning via Auxiliary Rotation Loss
• Distilling Localization for Self-Supervised Representation Learning
• Unsupervised Learning from Video with Deep Neural Embeddings
• Understand and Improve Contrastive Learning Methods for Visual Representation: A Review
• Temperature as Uncertainty in Contrastive Learning
• How Useful is Self-Supervised Pretraining for Visual Tasks?
• Category Contrast for Unsupervised Domain Adaptation in Visual Tasks
• Zero-Shot Learning from scratch (ZFS): leveraging local compositional representations
• DimCL: Dimensional Contrastive Learning For Improving Self-Supervised Learning
• UnShadowNet: Illumination Critic Guided Contrastive Learning For Shadow Removal
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• Understanding Hyperbolic Metric Learning through Hard Negative Sampling
• Leveraging Unlabelled Data in Multiple-Instance Learning Problems for Improved Detection of Parkinsonian Tremor in Free-Living Conditions
• Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoods
• Reviewing Evolution of Learning Functions and Semantic Information Measures for Understanding Deep Learning
• Representation Learning via Consistent Assignment of Views to Clusters
• Magnification-independent Histopathological Image Classification with Similarity-based Multi-scale Embeddings
• Spatial Ensemble: a Novel Model Smoothing Mechanism for Student-Teacher Framework
• Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning
• KinePose: A temporally optimized inverse kinematics technique for 6DOF human pose estimation with biomechanical constraints
• Prototypical Contrastive Learning through Alignment and Uniformity for Recommendation
• An empirical study of pretrained representations for few-shot classification
• EqCo: Equivalent Rules for Self-supervised Contrastive Learning
• AutoLTS: Automating Cycling Stress Assessment via Contrastive Learning and Spatial Post-processing
• Self-supervised Contrastive Learning for Irrigation Detection in Satellite Imagery
• Towards Demystifying Representation Learning with Non-contrastive Self-supervision
• Self-supervised Adversarial Training
• Tasting the cake: evaluating self-supervised generalization on out-of-distribution multimodal MRI data
• CoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and Denoising
• Pretrained Encoders are All You Need
• Representation Learning for Remote Sensing: An Unsupervised Sensor Fusion Approach
• Shuffle & Divide: Contrastive Learning for Long Text
• DTG-Net: Differentiated Teachers Guided Self-Supervised Video Action Recognition
• Self-Distilled Self-Supervised Representation Learning
• TVDIM: Enhancing Image Self-Supervised Pretraining via Noisy Text Data
• ContraCluster: Learning to Classify without Labels by Contrastive Self-Supervision and Prototype-Based Semi-Supervision
• CLAWS: Contrastive Learning with hard Attention and Weak Supervision
• Fast Development of ASR in African Languages using Self Supervised Speech Representation Learning
• CGCL: Collaborative Graph Contrastive Learning without Handcrafted Graph Data Augmentations
• A Novel Information-Theoretic Objective to Disentangle Representations for Fair Classification
• Synthesizing Photorealistic Images with Deep Generative Learning
• Transferrable Contrastive Learning for Visual Domain Adaptation
• Is High Variance Unavoidable in RL? A Case Study in Continuous Control
• Leveraging Hidden Structure in Self-Supervised Learning
• Graph Partner Neural Networks for Semi-Supervised Learning on Graphs
• Meta-Learning to Improve Pre-Training
• Transfer Learning of fMRI Dynamics
• An Image-based Approach of Task-driven Driving Scene Categorization
• Discriminative Clustering with Representation Learning with any Ratio of Labeled to Unlabeled Data
• InfoBehavior: Self-supervised Representation Learning for Ultra-long Behavior Sequence via Hierarchical Grouping