Mutual Information Gradient Estimation for Representation Learning

Abstract

Mutual Information (MI) plays an important role in representation learning. However, MI is unfortunately intractable in continuous and high-dimensional settings. Recent advances establish tractable and scalable MI estimators to discover useful representation. However, most of the existing methods are not capable of providing an accurate estimation of MI with low-variance when the MI is large. We argue that directly estimating the gradients of MI is more appealing for representation learning than estimating MI in itself. To this end, we propose the Mutual Information Gradient Estimator (MIGE) for representation learning based on the score estimation of implicit distributions. MIGE exhibits a tight and smooth gradient estimation of MI in the high-dimensional and large-MI settings. We expand the applications of MIGE in both unsupervised learning of deep representations based on InfoMax and the Information Bottleneck method. Experimental results have indicated significant performance improvement in learning useful representation.

ICLR 2020

References in corpus (15)

• Estimating Mutual Information
• The information bottleneck method
• Learning deep representations by mutual information estimation and maximization
• Opening the Black Box of Deep Neural Networks via Information
• Equitability, mutual information, and the maximal information coefficient
• MINE: Mutual Information Neural Estimation
• On Mutual Information Maximization for Representation Learning
• On Variational Bounds of Mutual Information
• Measuring Sample Quality with Stein's Method
• Sliced Score Matching: A Scalable Approach to Density and Score Estimation
• Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference
• Uncertainty in the Variational Information Bottleneck
• Formal Limitations on the Measurement of Mutual Information
• Gradient Estimators for Implicit Models
• A Spectral Approach to Gradient Estimation for Implicit Distributions

Cited by in corpus (3)

• Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization
• Bidirectional Generative Modeling Using Adversarial Gradient Estimation
• Nonparametric Score Estimators