A non-parametric k-nearest neighbour entropy estimator

Abstract

A non-parametric k-nearest neighbour based entropy estimator is proposed. It improves on the classical Kozachenko-Leonenko estimator by considering non-uniform probability densities in the region of k-nearest neighbours around each sample point. It aims at improving the classical estimators in three situations: first, when the dimensionality of the random variable is large; second, when near-functional relationships leading to high correlation between components of the random variable are present; and third, when the marginal variances of random variable components vary significantly with respect to each other. Heuristics on the error of the proposed and classical estimators are presented. Finally, the proposed estimator is tested for a variety of distributions in successively increasing dimensions and in the presence of a near-functional relationship. Its performance is compared with a classical estimator and shown to be a significant improvement.

Cited by in corpus (11)

• 400 Gbps Dual-polarisation Non-linear Frequency-division Multiplexed Transmission with b-Modulation
• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• Accurate estimation of the normalized mutual information of multidimensional data
• How does Weight Correlation Affect the Generalisation Ability of Deep Neural Networks
• Demystifying Fixed k-Nearest Neighbor Information Estimators
• Estimating Mutual Information by Local Gaussian Approximation
• DiversityGAN: Diversity-Aware Vehicle Motion Prediction via Latent Semantic Sampling
• Missing Features Reconstruction and Its Impact on Classification Accuracy
• Towards a robust approach to infer causality in molecular systems satisfying detailed balance
• Minimum-entropy constraints on galactic potentials
• Entropic Issues in Likelihood-Based OOD Detection