On the Estimation of Information Measures of Continuous Distributions

Abstract

The estimation of information measures of continuous distributions based on samples is a fundamental problem in statistics and machine learning. In this paper, we analyze estimates of differential entropy in $K$-dimensional Euclidean space, computed from a finite number of samples, when the probability density function belongs to a predetermined convex family $\mathcal{P}$. First, estimating differential entropy to any accuracy is shown to be infeasible if the differential entropy of densities in $\mathcal{P}$ is unbounded, clearly showing the necessity of additional assumptions. Subsequently, we investigate sufficient conditions that enable confidence bounds for the estimation of differential entropy. In particular, we provide confidence bounds for simple histogram based estimation of differential entropy from a fixed number of samples, assuming that the probability density function is Lipschitz continuous with known Lipschitz constant and known, bounded support. Our focus is on differential entropy, but we provide examples that show that similar results hold for mutual information and relative entropy as well.

20 pages

References in corpus (7)

• Representation Learning with Contrastive Predictive Coding
• Learning deep representations by mutual information estimation and maximization
• Opening the Black Box of Deep Neural Networks via Information
• Learning Discrete Representations via Information Maximizing Self-Augmented Training
• On Variational Bounds of Mutual Information
• Estimating Mutual Information for Discrete-Continuous Mixtures
• Formal Limitations on the Measurement of Mutual Information

Cited by in corpus (1)

• Neural Estimators for Conditional Mutual Information Using Nearest Neighbors Sampling