Estimating Mutual Information for Discrete-Continuous Mixtures

Abstract

Estimating mutual information from observed samples is a basic primitive, useful in several machine learning tasks including correlation mining, information bottleneck clustering, learning a Chow-Liu tree, and conditional independence testing in (causal) graphical models. While mutual information is a well-defined quantity in general probability spaces, existing estimators can only handle two special cases of purely discrete or purely continuous pairs of random variables. The main challenge is that these methods first estimate the (differential) entropies of X, Y and the pair (X;Y) and add them up with appropriate signs to get an estimate of the mutual information. These 3H-estimators cannot be applied in general mixture spaces, where entropy is not well-defined. In this paper, we design a novel estimator for mutual information of discrete-continuous mixtures. We prove that the proposed estimator is consistent. We provide numerical experiments suggesting superiority of the proposed estimator compared to other heuristics of adding small continuous noise to all the samples and applying standard estimators tailored for purely continuous variables, and quantizing the samples and applying standard estimators tailored for purely discrete variables. This significantly widens the applicability of mutual information estimation in real-world applications, where some variables are discrete, some continuous, and others are a mixture between continuous and discrete components.

25 pages, 3 figures. Part of this paper appears in the Conference on Neural Information Processing Systems (NIPS), 2017

References in corpus (3)

• Information Theoretical Estimators Toolbox
• Nonparanormal Information Estimation
• Ensemble estimators for multivariate entropy estimation

Cited by in corpus (23)

• Causal Confusion in Imitation Learning
• Understanding the Limitations of Variational Mutual Information Estimators
• Stochastic Mutual Information Gradient Estimation for Dimensionality Reduction Networks
• Information Flow in Computational Systems
• Information Theoretic Feature Transformation Learning for Brain Interfaces
• On Network Science and Mutual Information for Explaining Deep Neural Networks
• Inference of Probabilistic Programs with Moment-Matching Gaussian Mixtures
• Ensemble Estimation of Generalized Mutual Information with Applications to Genomics
• Discrete-Continuous Mixtures in Probabilistic Programming: Generalized Semantics and Inference Algorithms
• A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decomposition
• Minimax Optimal Nonparametric Estimation of Heterogeneous Treatment Effects
• On the Estimation of Information Measures of Continuous Distributions
• Can Information Flows Suggest Targets for Interventions in Neural Circuits?
• Random Forests for Adaptive Nearest Neighbor Estimation of Information-Theoretic Quantities
• A new Gini correlation between quantitative and qualitative variables
• Efficiently Estimating Mutual Information Between Attributes Across Tables
• Conditional Mutual Information Estimation for Mixed Discrete and Continuous Variables with Nearest Neighbors
• Interpretable Fault Detection using Projections of Mutual Information Matrix
• Non parametric estimation of joint, Renyi-Stallis entropies and mutual information and asymptotic limits
• Structure Learning of Markov Random Fields through Grow-Shrink Maximum Pseudolikelihood Estimation
• Model-Augmented Estimation of Conditional Mutual Information for Feature Selection
• GILBO: One Metric to Measure Them All
• On Privacy of Dynamical Systems: An Optimal Probabilistic Mapping Approach (Extended Preprint)