Ensemble Estimation of Information Divergence

Abstract

Recent work has focused on the problem of nonparametric estimation of information divergence functionals. Many existing approaches are restrictive in their assumptions on the density support set or require difficult calculations at the support boundary which must be known a priori. The MSE convergence rate of a leave-one-out kernel density plug-in divergence functional estimator for general bounded density support sets is derived where knowledge of the support boundary is not required. The theory of optimally weighted ensemble estimation is generalized to derive a divergence estimator that achieves the parametric rate when the densities are sufficiently smooth. The asymptotic distribution of this estimator and some guidelines for tuning parameter selection are provided. Based on the theory, an empirical estimator of Rényi-$\alpha$ divergence is proposed that outperforms the standard kernel density plug-in estimator, especially in high dimension. The estimator is shown to be robust to the choice of tuning parameters. As an illustration, we use the estimator to estimate bounds on the Bayes error rate of a classification problem.

27 pages, 4 figures; A previous version of this paper was posted under the title of "Improving Convergence of Divergence Functional Ensemble Estimators"

References in corpus (6)

• Does Dirichlet Prior Smoothing Solve the Shannon Entropy Estimation Problem?
• Maximum Likelihood Estimation of Functionals of Discrete Distributions
• Ensemble estimation of multivariate f-divergence
• The intrinsic value of HFO features as a biomarker of epileptic activity
• Exponential Concentration of a Density Functional Estimator
• Meta learning of bounds on the Bayes classifier error

Cited by in corpus (5)

• Ensemble Estimation of Generalized Mutual Information with Applications to Genomics
• Evaluating State-of-the-Art Classification Models Against Bayes Optimality
• Entropy Estimation via Uniformization
• Nearest neighbor density functional estimation from inverse Laplace transform
• Quantifying dependencies for sensitivity analysis with multivariate input sample data