Multivariate f-Divergence Estimation With Confidence

Abstract

The problem of f-divergence estimation is important in the fields of machine learning, information theory, and statistics. While several nonparametric divergence estimators exist, relatively few have known convergence properties. In particular, even for those estimators whose MSE convergence rates are known, the asymptotic distributions are unknown. We establish the asymptotic normality of a recently proposed ensemble estimator of f-divergence between two distributions from a finite number of samples. This estimator has MSE convergence rate of O(1/T), is simple to implement, and performs well in high dimensions. This theory enables us to perform divergence-based inference tasks such as testing equality of pairs of distributions based on empirical samples. We experimentally validate our theoretical results and, as an illustration, use them to empirically bound the best achievable classification error.

20 pages, 1 figure. Accepted to NIPS 2014 (supplementary material is included in the appendices)

References in corpus (3)

• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Ensemble estimation of multivariate f-divergence
• An information theoretic approach to the functional classification of neurons

Cited by in corpus (11)

• Information Theoretic Structure Learning with Confidence
• Direct estimation of density functionals using a polynomial basis
• On $w$-mixtures: Finite convex combinations of prescribed component distributions
• The intrinsic value of HFO features as a biomarker of epileptic activity
• Influence Functions for Machine Learning: Nonparametric Estimators for Entropies, Divergences and Mutual Informations
• Ensemble Estimation of Generalized Mutual Information with Applications to Genomics
• Evaluating State-of-the-Art Classification Models Against Bayes Optimality
• Practical and Consistent Estimation of f-Divergences
• Learning to Benchmark: Determining Best Achievable Misclassification Error from Training Data
• Neural Estimation of Statistical Divergences
• Image patch analysis of sunspots and active regions. II. Clustering via matrix factorization