Ensemble Estimation of Generalized Mutual Information with Applications to Genomics
arXiv:1701.08083 · doi:10.1109/TIT.2021.3100108
Abstract
Mutual information is a measure of the dependence between random variables that has been used successfully in myriad applications in many fields. Generalized mutual information measures that go beyond classical Shannon mutual information have also received much interest in these applications. We derive the mean squared error convergence rates of kernel density-based plug-in estimators of general mutual information measures between two multidimensional random variables and for two cases: 1) and are continuous; 2) and may have any mixture of discrete and continuous components. Using the derived rates, we propose an ensemble estimator of these information measures called GENIE by taking a weighted sum of the plug-in estimators with varied bandwidths. The resulting ensemble estimators achieve the parametric mean squared error convergence rate when the conditional densities of the continuous variables are sufficiently smooth. To the best of our knowledge, this is the first nonparametric mutual information estimator known to achieve the parametric convergence rate for the mixture case, which frequently arises in applications (e.g. variable selection in classification). The estimator is simple to implement and it uses the solution to an offline convex optimization problem and simple plug-in estimators. A central limit theorem is also derived for the ensemble estimators and minimax rates are derived for the continuous case. We demonstrate the ensemble estimator for the mixed case on simulated data and apply the proposed estimator to analyze gene relationships in single cell data.
arXiv commentPublished in IEEE Transactions on Information Theory in 2021; 42 pages, 3 figures; a shorter version of this paper was published at IEEE ISIT 2017 under the title "Ensemble estimation of mutual information"
References in corpus (28)
- Estimating Mutual Information
- A Review of Feature Selection Methods Based on Mutual Information
- Min- and Max- Relative Entropies and a New Entanglement Monotone
- The Gravity Dual of Renyi Entropy
- Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
- Aggregation for Gaussian regression
- Maximum Likelihood Estimation of Functionals of Discrete Distributions
- Does Dirichlet Prior Smoothing Solve the Shannon Entropy Estimation Problem?
- Estimating Mutual Information for Discrete-Continuous Mixtures
- Efficient Estimation of Mutual Information for Strongly Dependent Variables
- Ensemble estimation of multivariate f-divergence
- Operational Interpretation of Renyi Information Measures via Composite Hypothesis Testing Against Product and Markov Distributions
- Universal and Composite Hypothesis Testing via Mismatched Divergence
- Improved Lower Bounds for Constant GC-Content DNA Codes
- Changing the Environment Based on Empowerment as Intrinsic Motivation
- Multivariate f-Divergence Estimation With Confidence
- Direct Estimation of Information Divergence Using Nearest Neighbor Ratios
- An information theoretic approach to the functional classification of neurons
- Ensemble Estimation of Information Divergence
- Demystifying Fixed k-Nearest Neighbor Information Estimators
- Information Theoretic Structure Learning with Confidence
- The intrinsic value of HFO features as a biomarker of epileptic activity
- Two-stage Sampling, Prediction and Adaptive Regression via Correlation Screening (SPARCS)
- Meta learning of bounds on the Bayes classifier error
- Empirically Estimable Classification Bounds Based on a New Divergence Measure
- Nearest neighbor density functional estimation from inverse Laplace transform
- Ensemble Estimation of Distributional Functionals via -Nearest Neighbors
- Distributed High Dimensional Information Theoretical Image Registration via Random Projections