Formal Limitations on the Measurement of Mutual Information

Abstract

Measuring mutual information from finite data is difficult. Recent work has considered variational methods maximizing a lower bound. In this paper, we prove that serious statistical limitations are inherent to any method of measuring mutual information. More specifically, we show that any distribution-free high-confidence lower bound on mutual information estimated from N samples cannot be larger than O(ln N ).

References in corpus (8)

• Representation Learning with Contrastive Predictive Coding
• Estimating Mutual Information
• The information bottleneck method
• Learning deep representations by mutual information estimation and maximization
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• MINE: Mutual Information Neural Estimation
• On Variational Bounds of Mutual Information
• Information Theoretic Co-Training

Cited by in corpus (34)

• Learning Representations by Maximizing Mutual Information Across Views
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• On Mutual Information Maximization for Representation Learning
• Dream to Control: Learning Behaviors by Latent Imagination
• Rethinking Softmax with Cross-Entropy: Neural Network Classifier as Mutual Information Estimator
• Understanding the Limitations of Variational Mutual Information Estimators
• CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP
• Unsupervised State Representation Learning in Atari
• GraphCL: Contrastive Self-Supervised Learning of Graph Representations
• A Divergence Minimization Perspective on Imitation Learning Methods
• Image Clustering using an Augmented Generative Adversarial Network and Information Maximization
• Wasserstein Dependency Measure for Representation Learning
• Conditional Mutual Information Neural Estimator
• Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization
• Bottleneck Problems: Information and Estimation-Theoretic View
• Unsupervised Discovery of Temporal Structure in Noisy Data with Dynamical Components Analysis
• Neural Estimators for Conditional Mutual Information Using Nearest Neighbors Sampling
• Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
• Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• Mutual Information Gradient Estimation for Representation Learning
• Data-Efficient Mutual Information Neural Estimator
• A Plug-in Method for Representation Factorization in Connectionist Models
• Telescoping Density-Ratio Estimation
• Learning Efficient Representation for Intrinsic Motivation
• On the Estimation of Information Measures of Continuous Distributions
• On Predictive Information in RNNs
• KLDivNet: An unsupervised neural network for multi-modality image registration
• Efficient Empowerment Estimation for Unsupervised Stabilization
• TVDIM: Enhancing Image Self-Supervised Pretraining via Noisy Text Data
• Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap
• Neural Network Classifier as Mutual Information Evaluator
• InfoBehavior: Self-supervised Representation Learning for Ultra-long Behavior Sequence via Hierarchical Grouping