On Mutual Information Maximization for Representation Learning

Abstract

Many recent methods for unsupervised or self-supervised representation learning train feature extractors by maximizing an estimate of the mutual information (MI) between different views of the data. This comes with several immediate problems: For example, MI is notoriously hard to estimate, and using it as an objective for representation learning may lead to highly entangled representations due to its invariance under arbitrary invertible transformations. Nevertheless, these methods have been repeatedly shown to excel in practice. In this paper we argue, and provide empirical evidence, that the success of these methods cannot be attributed to the properties of MI alone, and that they strongly depend on the inductive bias in both the choice of feature extractor architectures and the parametrization of the employed MI estimators. Finally, we establish a connection to deep metric learning and argue that this interpretation may be a plausible explanation for the success of the recently introduced methods.

arXiv commentICLR 2020. Michael Tschannen and Josip Djolonga contributed equally

References in corpus (18)

• Scikit-learn: Machine Learning in Python
• Auto-Encoding Variational Bayes
• Representation Learning with Contrastive Predictive Coding
• Estimating Mutual Information
• Learning deep representations by mutual information estimation and maximization
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Density estimation using Real NVP
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Contrastive Multiview Coding
• Recent Advances in Autoencoder-Based Representation Learning
• Learning Discrete Representations via Information Maximizing Self-Augmented Training
• Learning Video Representations using Contrastive Bidirectional Transformer
• On Variational Bounds of Mutual Information
• Revisiting Self-Supervised Visual Representation Learning
• Formal Limitations on the Measurement of Mutual Information
• A Theory of Usable Information Under Computational Constraints
• Tighter Variational Representations of f-Divergences via Restriction to Probability Measures
• Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency

Cited by in corpus (97)

• A Simple Framework for Contrastive Learning of Visual Representations
• Graph Contrastive Learning with Adaptive Augmentation
• Contrastive Representation Learning: A Framework and Review
• What Makes for Good Views for Contrastive Learning?
• Graph Self-Supervised Learning: A Survey
• Prototypical Contrastive Learning of Unsupervised Representations
• Hard Negative Mixing for Contrastive Learning
• A survey on Semi-, Self- and Unsupervised Learning for Image Classification
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Pre-training Molecular Graph Representation with 3D Geometry
• Supervised Contrastive Learning
• Learning Robust Representations via Multi-View Information Bottleneck
• A Mutual Information Maximization Perspective of Language Representation Learning
• Maximum-Entropy Adversarial Data Augmentation for Improved Generalization and Robustness
• Intriguing Properties of Contrastive Losses
• Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
• CoMIR: Contrastive Multimodal Image Representation for Registration
• Contrastive Learning with Adversarial Examples
• Graph Clustering with Graph Neural Networks
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• Whitening for Self-Supervised Representation Learning
• Automatic Shortcut Removal for Self-Supervised Representation Learning
• Representation Learning via Invariant Causal Mechanisms
• Understanding the Limitations of Variational Mutual Information Estimators
• How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?
• Self-Supervised Relational Reasoning for Representation Learning
• Contrastive Learning Inverts the Data Generating Process
• Conditional Negative Sampling for Contrastive Learning of Visual Representations
• GraphCL: Contrastive Self-Supervised Learning of Graph Representations
• TCL: Transformer-based Dynamic Graph Modelling via Contrastive Learning
• Cross-Domain Few-Shot Learning by Representation Fusion
• Action and Perception as Divergence Minimization
• Neural Transformation Learning for Deep Anomaly Detection Beyond Images
• Self-Supervised Learning with Kernel Dependence Maximization
• Towards the Generalization of Contrastive Self-Supervised Learning
• Self-supervised EEG Representation Learning for Automatic Sleep Staging
• Better Fine-Tuning by Reducing Representational Collapse
• Learning Optimal Representations with the Decodable Information Bottleneck
• Unsupervised Two-Stage Anomaly Detection
• Contrastive learning, multi-view redundancy, and linear models
• Multi-Scale Contrastive Siamese Networks for Self-Supervised Graph Representation Learning
• A case for new neural network smoothness constraints
• Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
• Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination
• Global-Local Bidirectional Reasoning for Unsupervised Representation Learning of 3D Point Clouds
• KNN-BERT: Fine-Tuning Pre-Trained Models with KNN Classifier
• Self-supervised Learning from a Multi-view Perspective
• Unsupervised Deep Metric Learning via Auxiliary Rotation Loss
• Iterative Graph Self-Distillation
• Category Contrast for Unsupervised Domain Adaptation in Visual Tasks
• Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization
• Run Away From your Teacher: Understanding BYOL by a Novel Self-Supervised Approach
• Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning
• Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck
• Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction
• Evolution Is All You Need: Phylogenetic Augmentation for Contrastive Learning
• RGB-D Saliency Detection via Cascaded Mutual Information Minimization
• Mutual Information Gradient Estimation for Representation Learning
• Renormalized Mutual Information for Artificial Scientific Discovery
• Neural Methods for Point-wise Dependency Estimation
• Reviewing Evolution of Learning Functions and Semantic Information Measures for Understanding Deep Learning
• Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning
• Personalised Federated Learning On Heterogeneous Feature Spaces
• Dual Contrastive Loss and Attention for GANs
• DHOG: Deep Hierarchical Object Grouping
• Contrastive Distillation on Intermediate Representations for Language Model Compression
• Self-supervised Representation Learning with Relative Predictive Coding
• A Semantic Generalization of Shannon's Information Theory and Applications
• EqCo: Equivalent Rules for Self-supervised Contrastive Learning
• Function Contrastive Learning of Transferable Meta-Representations
• Mapping in a cycle: Sinkhorn regularized unsupervised learning for point cloud shapes
• Learnt dynamics generalizes across tasks, datasets, and populations
• ProtoInfoMax: Prototypical Networks with Mutual Information Maximization for Out-of-Domain Detection
• Conditional Mutual information-based Contrastive Loss for Financial Time Series Forecasting
• Contrastive Attraction and Contrastive Repulsion for Representation Learning
• Self-supervised Adversarial Training
• Functional Regularization for Representation Learning: A Unified Theoretical Perspective
• Shuffle & Divide: Contrastive Learning for Long Text
• Efficient Empowerment Estimation for Unsupervised Stabilization
• Addressing Feature Suppression in Unsupervised Visual Representations
• Representation Learning for Sequence Data with Deep Autoencoding Predictive Components
• On Local Aggregation in Heterophilic Graphs
• Hard Negative Sampling via Regularized Optimal Transport for Contrastive Representation Learning
• Interest-Behaviour Multiplicative Network for Resource-limited Recommendation
• ContraCluster: Learning to Classify without Labels by Contrastive Self-Supervision and Prototype-Based Semi-Supervision
• Detection of Abnormal Behavior with Self-Supervised Gaze Estimation
• Recent Advancements in Self-Supervised Paradigms for Visual Feature Representation
• Pretext Tasks selection for multitask self-supervised speech representation learning
• On Sampling-Based Training Criteria for Neural Language Modeling
• Explaining Representation by Mutual Information
• Cross-modal Image Retrieval with Deep Mutual Information Maximization
• Contrastive Representation Learning with Trainable Augmentation Channel
• Unsupervised Representation Learning via Neural Activation Coding
• Encoding Causal Macrovariables
• Transfer Learning of fMRI Dynamics