On Variational Bounds of Mutual Information

Abstract

Estimating and optimizing Mutual Information (MI) is core to many problems in machine learning; however, bounding MI in high dimensions is challenging. To establish tractable and scalable objectives, recent work has turned to variational bounds parameterized by neural networks, but the relationships and tradeoffs between these bounds remains unclear. In this work, we unify these recent developments in a single framework. We find that the existing variational lower bounds degrade when the MI is large, exhibiting either high bias or high variance. To address this problem, we introduce a continuum of lower bounds that encompasses previous bounds and flexibly trades off bias and variance. On high-dimensional, controlled problems, we empirically characterize the bias and variance of the bounds and their gradients and demonstrate the effectiveness of our new bounds for estimation and representation learning.

arXiv commentICML 2019

Cited by in corpus (81)

• Graph Contrastive Learning with Adaptive Augmentation
• Contrastive Representation Learning: A Framework and Review
• Learning Representations by Maximizing Mutual Information Across Views
• Contrastive Multiview Coding
• On Mutual Information Maximization for Representation Learning
• A survey on Semi-, Self- and Unsupervised Learning for Image Classification
• Pre-training Molecular Graph Representation with 3D Geometry
• Adversarial Graph Augmentation to Improve Graph Contrastive Learning
• Graph Information Bottleneck
• Learning and Evaluating Representations for Deep One-class Classification
• Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
• Formal Limitations on the Measurement of Mutual Information
• CoMIR: Contrastive Multimodal Image Representation for Registration
• Restricting the Flow: Information Bottlenecks for Attribution
• On Information Plane Analyses of Neural Network Classifiers -- A Review
• Normalization of breast MRIs using Cycle-Consistent Generative Adversarial Networks
• Representation Learning via Invariant Causal Mechanisms
• Rethinking Softmax with Cross-Entropy: Neural Network Classifier as Mutual Information Estimator
• Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' Outputs
• Unsupervised State Representation Learning in Atari
• Disentangling Label Distribution for Long-tailed Visual Recognition
• Predictive Information Accelerates Learning in RL
• Joint Contrastive Learning for Unsupervised Domain Adaptation
• Mutual Information Scaling for Tensor Network Machine Learning
• BYOL works even without batch statistics
• Mutual information, quantum phase transition, and phase coherence in Kondo systems
• Shape or Texture: Understanding Discriminative Features in CNNs
• Hyperparameter Auto-tuning in Self-Supervised Robotic Learning
• Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identification
• A Unified Stochastic Gradient Approach to Designing Bayesian-Optimal Experiments
• M2IOSR: Maximal Mutual Information Open Set Recognition
• Temperature as Uncertainty in Contrastive Learning
• Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization
• Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
• Neural Estimators for Conditional Mutual Information Using Nearest Neighbors Sampling
• Causal Influence Detection for Improving Efficiency in Reinforcement Learning
• Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction
• Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning
• On the Fairness of Disentangled Representations
• Usable Information and Evolution of Optimal Representations During Training
• Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Design
• Learning Discrete Structured Representations by Adversarially Maximizing Mutual Information
• Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoods
• Mutual Information Gradient Estimation for Representation Learning
• Towards Empirical Sandwich Bounds on the Rate-Distortion Function
• Contrastive Unsupervised Learning for Speech Emotion Recognition
• Telescoping Density-Ratio Estimation
• DEMI: Discriminative Estimator of Mutual Information
• CLCC: Contrastive Learning for Color Constancy
• Discriminative Mutual Information Estimators for Channel Capacity Learning
• Improving Disentangled Text Representation Learning with Information-Theoretic Guidance
• On the Estimation of Information Measures of Continuous Distributions
• On Predictive Information in RNNs
• DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneck
• Featurized Density Ratio Estimation
• Density Ratio Estimation via Infinitesimal Classification
• An Empirical Study of Graph Contrastive Learning
• Dual Contradistinctive Generative Autoencoder
• Conditional Mutual information-based Contrastive Loss for Financial Time Series Forecasting
• Compressive Visual Representations
• Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence
• SentenceMIM: A Latent Variable Language Model
• C-MI-GAN : Estimation of Conditional Mutual Information using MinMax formulation
• Likelihood Ratio Exponential Families
• Memory Augmented Multi-Instance Contrastive Predictive Coding for Sequential Recommendation
• Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learning
• Cross-domain Imitation from Observations
• Reliable Estimation of KL Divergence using a Discriminator in Reproducing Kernel Hilbert Space
• Neural Computation of Capacity Region of Memoryless Multiple Access Channels
• Disentangled Variational Information Bottleneck for Multiview Representation Learning
• Neural Network Classifier as Mutual Information Evaluator
• On Study of Mutual Information and its Estimation Methods
• Stein Latent Optimization for Generative Adversarial Networks
• On the Limitations of Multimodal VAEs
• Distiller: A Systematic Study of Model Distillation Methods in Natural Language Processing
• Deep Unsupervised Image Anomaly Detection: An Information Theoretic Framework
• Contrastive Active Inference
• On Sampling-Based Training Criteria for Neural Language Modeling
• Follow the Neurally-Perturbed Leader for Adversarial Training
• Stochastic Contrastive Learning
• Model-Augmented Estimation of Conditional Mutual Information for Feature Selection