Ensemble estimation of multivariate f-divergence

Abstract

f-divergence estimation is an important problem in the fields of information theory, machine learning, and statistics. While several divergence estimators exist, relatively few of their convergence rates are known. We derive the MSE convergence rate for a density plug-in estimator of f-divergence. Then by applying the theory of optimally weighted ensemble estimation, we derive a divergence estimator with a convergence rate of O(1/T) that is simple to implement and performs well in high dimensions. We validate our theoretical results with experiments.

14 pages, 6 figures, a condensed version of this paper was accepted to ISIT 2014, Version 2: Moved the proofs of the theorems from the main body to appendices at the end

References in corpus (2)

• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Ensemble estimation of multivariate f-divergence

Cited by in corpus (27)

• Efficient Estimation of Mutual Information for Strongly Dependent Variables
• Ensemble estimation of multivariate f-divergence
• Multivariate f-Divergence Estimation With Confidence
• Improving Zero-shot Voice Style Transfer via Disentangled Representation Learning
• Direct Estimation of Information Divergence Using Nearest Neighbor Ratios
• Analysis of k-Nearest Neighbor Distances with Application to Entropy Estimation
• Ensemble Estimation of Information Divergence
• Demystifying Fixed k-Nearest Neighbor Information Estimators
• Importance of Small Probability Events in Big Data: Information Measures, Applications, and Challenges
• Information Theoretic Structure Learning with Confidence
• The intrinsic value of HFO features as a biomarker of epileptic activity
• Ensemble Estimation of Generalized Mutual Information with Applications to Genomics
• Exponential Concentration of a Density Functional Estimator
• Meta learning of bounds on the Bayes classifier error
• Estimating Mutual Information by Local Gaussian Approximation
• Estimation of KL Divergence: Optimal Minimax Rate
• Finite-Sample Analysis of Fixed-k Nearest Neighbor Density Functional Estimators
• Nearest neighbor density functional estimation from inverse Laplace transform
• Practical and Consistent Estimation of f-Divergences
• Minimax Optimal Estimation of KL Divergence for Continuous Distributions
• Ensemble Estimation of Distributional Functionals via $k$-Nearest Neighbors
• Divergence Measures Estimation and Its Asymptotic Normality Theory Using Wavelets Empirical Processes
• Kullback-Leibler Divergence-Based Out-of-Distribution Detection with Flow-Based Generative Models
• Efficient Nonparametric Smoothness Estimation
• Divergence measures estimation and its asymptotic normality theory in the discrete case
• Image patch analysis of sunspots and active regions. II. Clustering via matrix factorization
• Consistency Bands for divergences measures