On Variational Bounds of Mutual Information

Abstract

Estimating and optimizing Mutual Information (MI) is core to many problems in machine learning; however, bounding MI in high dimensions is challenging. To establish tractable and scalable objectives, recent work has turned to variational bounds parameterized by neural networks, but the relationships and tradeoffs between these bounds remains unclear. In this work, we unify these recent developments in a single framework. We find that the existing variational lower bounds degrade when the MI is large, exhibiting either high bias or high variance. To address this problem, we introduce a continuum of lower bounds that encompasses previous bounds and flexibly trades off bias and variance. On high-dimensional, controlled problems, we empirically characterize the bias and variance of the bounds and their gradients and demonstrate the effectiveness of our new bounds for estimation and representation learning.

ICML 2019

Cited by in corpus (32)

• Contrastive Representation Learning: A Framework and Review
• Learning Representations by Maximizing Mutual Information Across Views
• Adversarial Graph Augmentation to Improve Graph Contrastive Learning
• Learning and Evaluating Representations for Deep One-class Classification
• Disentangling Label Distribution for Long-tailed Visual Recognition
• Joint Contrastive Learning for Unsupervised Domain Adaptation
• Mutual information, quantum phase transition, and phase coherence in Kondo systems
• Shape or Texture: Understanding Discriminative Features in CNNs
• A Unified Stochastic Gradient Approach to Designing Bayesian-Optimal Experiments
• Temperature as Uncertainty in Contrastive Learning
• M2IOSR: Maximal Mutual Information Open Set Recognition
• Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
• Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoods
• Mutual Information Gradient Estimation for Representation Learning
• Contrastive Unsupervised Learning for Speech Emotion Recognition
• DEMI: Discriminative Estimator of Mutual Information
• Discriminative Mutual Information Estimators for Channel Capacity Learning
• CLCC: Contrastive Learning for Color Constancy
• On Predictive Information in RNNs
• An Empirical Study of Graph Contrastive Learning
• Dual Contradistinctive Generative Autoencoder
• Featurized Density Ratio Estimation
• Memory Augmented Multi-Instance Contrastive Predictive Coding for Sequential Recommendation
• Likelihood Ratio Exponential Families
• C-MI-GAN : Estimation of Conditional Mutual Information using MinMax formulation
• Cross-domain Imitation from Observations
• Reliable Estimation of KL Divergence using a Discriminator in Reproducing Kernel Hilbert Space
• On Study of Mutual Information and its Estimation Methods
• Neural Network Classifier as Mutual Information Evaluator
• Distiller: A Systematic Study of Model Distillation Methods in Natural Language Processing
• Disentangled Variational Information Bottleneck for Multiview Representation Learning
• Deep Unsupervised Image Anomaly Detection: An Information Theoretic Framework