Estimating the Mutual Information between two Discrete, Asymmetric Variables with Limited Samples

Abstract

Determining the strength of non-linear statistical dependencies between two variables is a crucial matter in many research fields. The established measure for quantifying such relations is the mutual information. However, estimating mutual information from limited samples is a challenging task. Since the mutual information is the difference of two entropies, the existing Bayesian estimators of entropy may be used to estimate information. This procedure, however, is still biased in the severely under-sampled regime. Here we propose an alternative estimator that is applicable to those cases in which the marginal distribution of one of the two variables---the one with minimal entropy---is well sampled. The other variable, as well as the joint and conditional distributions, can be severely undersampled. We obtain an estimator that presents very low bias, outperforming previous methods even when the sampled data contain few coincidences. As with other Bayesian estimators, our proposal focuses on the strength of the interaction between two discrete variables, without seeking to model the specific way in which the variables are related. A distinctive property of our method is that the main data statistics determining the amount of mutual information is the inhomogeneity of the conditional distribution of the low-entropy variable in those states (typically few) in which the large-entropy variable registers coincidences.

References in corpus (18)

• Estimating Mutual Information
• The information bottleneck method
• Opening the Black Box of Deep Neural Networks via Information
• Equitability, mutual information, and the maximal information coefficient
• MINE: Mutual Information Neural Estimation
• Quantifying unique information
• Nonnegative Decomposition of Multivariate Information
• Entropy and information in neural spike trains: Progress on the sampling problem
• A Bivariate Measure of Redundant Information
• Measuring multivariate redundant information with pointwise common change in surprisal
• Bias Analysis in Entropy Estimation
• Estimating Mixture Entropy with Pairwise Distances
• Inference of entropies of discrete random variables with unknown cardinalities
• Temporal correlations and neural spike train entropy
• Information estimation using nonparametric copulas
• Estimating probabilities from experimental frequencies
• Estimating Functions of Distributions Defined over Spaces of Unknown Size
• Information-theoretical analysis of the statistical dependencies among three variables: Applications to written language

Cited by in corpus (5)

• Optimal Renormalization Group Transformation from Information Theory
• The Illusion of Randomness: An Empirical Analysis of Address Space Layout Randomization Implementations
• Low probability states, data statistics, and entropy estimation
• String amplitudes and mutual information in confining backgrounds: the partonic behavior
• Inferring a property of a large system from a small number of samples