Understanding the Limitations of Variational Mutual Information Estimators

Abstract

Variational approaches based on neural networks are showing promise for estimating mutual information (MI) between high dimensional variables. However, they can be difficult to use in practice due to poorly understood bias/variance tradeoffs. We theoretically show that, under some conditions, estimators such as MINE exhibit variance that could grow exponentially with the true amount of underlying MI. We also empirically demonstrate that existing estimators fail to satisfy basic self-consistency properties of MI, such as data processing and additivity under independence. Based on a unified perspective of variational approaches, we develop a new estimator that focuses on variance reduction. Empirical results on standard benchmark tasks demonstrate that our proposed estimator exhibits improved bias-variance trade-offs on standard benchmark tasks.

arXiv commentFixed some typos, credit to Yilun Xu

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Representation Learning with Contrastive Predictive Coding
• Learning deep representations by mutual information estimation and maximization
• NICE: Non-linear Independent Components Estimation
• Density estimation using Real NVP
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
• MINE: Mutual Information Neural Estimation
• Deep Variational Information Bottleneck
• Towards a Definition of Disentangled Representations
• On Mutual Information Maximization for Representation Learning
• Entropy and information in neural spike trains: Progress on the sampling problem
• Estimating Mutual Information for Discrete-Continuous Mixtures
• Bias and Generalization in Deep Generative Models: An Empirical Study
• Formal Limitations on the Measurement of Mutual Information
• The Information Autoencoding Family: A Lagrangian Perspective on Latent Variable Generative Models
• CCMI : Classifier based Conditional Mutual Information Estimation

Cited by in corpus (16)

• Intriguing Properties of Contrastive Losses
• Bottleneck Problems: Information and Estimation-Theoretic View
• Neural Estimators for Conditional Mutual Information Using Nearest Neighbors Sampling
• Multi-label Contrastive Predictive Coding
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• Neural Methods for Point-wise Dependency Estimation
• Telescoping Density-Ratio Estimation
• A Neural Network MCMC sampler that maximizes Proposal Entropy
• Conditional Mutual information-based Contrastive Loss for Financial Time Series Forecasting
• Featurized Density Ratio Estimation
• Information Condensing Active Learning
• Neural Dependency Coding inspired Multimodal Fusion
• On Study of Mutual Information and its Estimation Methods
• Recent Advancements in Self-Supervised Paradigms for Visual Feature Representation
• A Hybrid Gradient Method to Designing Bayesian Experiments for Implicit Models
• Analysis of Discriminator in RKHS Function Space for Kullback-Leibler Divergence Estimation