Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization

Abstract

Successful applications of InfoNCE and its variants have popularized the use of contrastive variational mutual information (MI) estimators in machine learning. While featuring superior stability, these estimators crucially depend on costly large-batch training, and they sacrifice bound tightness for variance reduction. To overcome these limitations, we revisit the mathematics of popular variational MI bounds from the lens of unnormalized statistical modeling and convex optimization. Our investigation not only yields a new unified theoretical framework encompassing popular variational MI bounds but also leads to a novel, simple, and powerful contrastive MI estimator named as FLO. Theoretically, we show that the FLO estimator is tight, and it provably converges under stochastic gradient descent. Empirically, our FLO estimator overcomes the limitations of its predecessors and learns more efficiently. The utility of FLO is verified using an extensive set of benchmarks, which also reveals the trade-offs in practical MI estimation.

References in corpus (15)

• Auto-Encoding Variational Bayes
• Learning Transferable Visual Models From Natural Language Supervision
• Estimating Mutual Information
• Equality of Opportunity in Supervised Learning
• Learning deep representations by mutual information estimation and maximization
• Opening the Black Box of Deep Neural Networks via Information
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Contrastive Multiview Coding
• Deep Variational Information Bottleneck
• On Mutual Information Maximization for Representation Learning
• AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias
• On Variational Bounds of Mutual Information
• Formal Limitations on the Measurement of Mutual Information
• Learning Controllable Fair Representations
• Mutual Information Gradient Estimation for Representation Learning

Cited by in corpus (4)

• Variational Bayesian experimental design for geophysical applications: seismic source location, amplitude versus offset inversion, and estimating CO2 saturations in a subsurface reservoir
• Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
• Supercharging Imbalanced Data Learning With Energy-based Contrastive Representation Transfer
• Quantifying the Task-Specific Information in Text-Based Classifications