Efficient Estimation of Mutual Information for Strongly Dependent Variables
arXiv:1411.2003
Abstract
We demonstrate that a popular class of nonparametric mutual information (MI) estimators based on k-nearest-neighbor graphs requires number of samples that scales exponentially with the true MI. Consequently, accurate estimation of MI between two strongly dependent variables is possible only for prohibitively large sample size. This important yet overlooked shortcoming of the existing estimators is due to their implicit reliance on local uniformity of the underlying joint distribution. We introduce a new estimator that is robust to local non-uniformity, works well with limited data, and is able to capture relationship strengths over many orders of magnitude. We demonstrate the superior performance of the proposed estimator on both synthetic and real-world data.
13 pages, to appear in International Conference on Artificial Intelligence and Statistics (AISTATS) 2015
References in corpus (7)
- Estimating Mutual Information
- Equitability, mutual information, and the maximal information coefficient
- Principal Manifolds and Nonlinear Dimension Reduction via Local Tangent Space Alignment
- A consistent multivariate test of association based on ranks of distances
- Information Theoretical Estimators Toolbox
- Comment on "Detecting Novel Associations In Large Data Sets" by Reshef Et Al, Science Dec 16, 2011
- Ensemble estimation of multivariate f-divergence
Cited by in corpus (10)
- Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
- Information Perspective to Probabilistic Modeling: Boltzmann Machines versus Born Machines
- Demystifying Fixed k-Nearest Neighbor Information Estimators
- Inflation as an Information Bottleneck - A strategy for identifying universality classes and making robust predictions
- Ensemble Estimation of Generalized Mutual Information with Applications to Genomics
- Geometric Estimation of Multivariate Dependency
- Quantifying dependencies for sensitivity analysis with multivariate input sample data
- Impact of Bit Allocation Strategies on Machine Learning Performance in Rate Limited Systems
- Efficiently Estimating Mutual Information Between Attributes Across Tables
- Optimizing Privacy and Utility Tradeoffs for Group Interests Through Harmonization