A class of Rényi information estimators for multidimensional densities

Abstract

A class of estimators of the Rényi and Tsallis entropies of an unknown distribution $f$ in ${\mathbb{R}}^m$ is presented. These estimators are based on the $k$th nearest-neighbor distances computed from a sample of $N$ i.i.d. vectors with distribution $f$. We show that entropies of any order $q$, including Shannon's entropy, can be estimated consistently with minimal assumptions on $f$. Moreover, we show that it is straightforward to extend the nearest-neighbor method to estimate the statistical distance between two distributions using one i.i.d. sample from each. (Wit Correction.)

Published in at http://dx.doi.org/10.1214/07-AOS539 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Cited by in corpus (47)

• Detecting causal associations in large nonlinear time series datasets
• The relation between Granger causality and directed information theory: a review
• On directed information theory and Granger causality graphs
• Estimation of Rényi Entropy and Mutual Information Based on Generalized Nearest-Neighbor Graphs
• On Rényi and Tsallis entropies and divergences for exponential families
• A closed-form expression for the Sharma-Mittal entropy of exponential families
• Nonparametric Divergence Estimation with Applications to Machine Learning on Distributions
• HRel: Filter Pruning based on High Relevance between Activation Maps and Class Labels
• A Kullback-Leibler divergence measure of intermittency: application to turbulence
• Conditional independence testing based on a nearest-neighbor estimator of conditional mutual information
• Entropic measures of Rydberg-like harmonic states
• Rényi entropies of the highly-excited states of multidimensional harmonic oscillators by use of strong Laguerre asymptotics
• Limit theory for point processes in manifolds
• Entropic properties of $D$-dimensional Rydberg systems
• Entropic uncertainty measures for large dimensional hydrogenic systems
• Analysis of k-Nearest Neighbor Distances with Application to Entropy Estimation
• Empirical estimation of entropy functionals with confidence
• Complexity measures and uncertainty relations of the high-dimensional harmonic and hydrogenic systems
• Entropic one-class classifiers
• Mixing Bandt-Pompe and Lempel-Ziv approaches: another way to analyze the complexity of continuous-states sequences
• Analytical Shannon information entropies for all discrete multidimensional hydrogenic states
• Heisenberg and entropic uncertainty measures for large-dimensional harmonic systems
• Kernels on Sample Sets via Nonparametric Divergence Estimates
• Rényi entropies for multidimensional hydrogenic systems in position and momentum spaces
• Exact Rényi entropies of $D$-dimensional harmonic systems
• Direct estimation of density functionals using a polynomial basis
• Entropy and complexity analysis of the D-dimensional rigid rotator and hyperspherical harmonics
• Exponential Concentration of a Density Functional Estimator
• Comparing and weighting imperfect models using D-probabilities
• infomeasure: A Comprehensive Python Package for Information Theory Measures and Estimators
• Ensemble estimators for multivariate entropy estimation
• Geometric Estimation of Multivariate Dependency
• Finite-Sample Analysis of Fixed-k Nearest Neighbor Density Functional Estimators
• Nearest neighbor density functional estimation from inverse Laplace transform
• One-class classifiers based on entropic spanning graphs
• On the analysis of signals in a permutation Lempel-Ziv complexity - permutation Shannon entropy plane
• Chaos in violent relaxation dynamics. Disentangling micro- and macro-chaos in numerical experiments of dissipationless collapse
• Minimum-entropy constraints on galactic potentials
• On mutual information estimation for mixed-pair random variables
• Distributed High Dimensional Information Theoretical Image Registration via Random Projections
• Plans D'Experiences D'Information De Kullback-Leibler Minimale
• Efficient Nonparametric Smoothness Estimation
• Statistical tests based on Rényi entropy estimation
• Causal inference in time series in terms of Rényi transfer entropy
• co-BPM: a Bayesian Model for Divergence Estimation
• Dimension Estimation Using Random Connection Models
• Limit Theorems for Empirical Rényi Entropy and Divergence with Applications to Molecular Diversity Analysis