Exponential Concentration of a Density Functional Estimator

Abstract

We analyze a plug-in estimator for a large class of integral functionals of one or more continuous probability densities. This class includes important families of entropy, divergence, mutual information, and their conditional versions. For densities on the $d$-dimensional unit cube $[0,1{]}^d$ that lie in a $\beta$-Hölder smoothness class, we prove our estimator converges at the rate $O\left(n^{-\frac{\beta }{\beta +d}}\right)$. Furthermore, we prove the estimator is exponentially concentrated about its mean, whereas most previous related results have proven only expected error bounds on estimators.

arXiv commentIn 29th Annual Conference on Neural Information Processing Systems (NIPS), 2014

References in corpus (5)

• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Kernel-based Conditional Independence Test and Application in Causal Discovery
• A class of Rényi information estimators for multidimensional densities
• Ensemble estimation of multivariate f-divergence
• Undercomplete Blind Subspace Deconvolution

Cited by in corpus (6)

• Analysis of k-Nearest Neighbor Distances with Application to Entropy Estimation
• Ensemble Estimation of Information Divergence
• Demystifying Fixed k-Nearest Neighbor Information Estimators
• Information Theoretic Structure Learning with Confidence
• Finite-Sample Analysis of Fixed-k Nearest Neighbor Density Functional Estimators
• Ensemble Estimation of Distributional Functionals via $k$-Nearest Neighbors