On the consistency of the Kozachenko-Leonenko entropy estimate

Abstract

We revisit the problem of the estimation of the differential entropy $H(f)$ of a random vector $X$ in $R^d$ with density $f$, assuming that $H(f)$ exists and is finite. In this note, we study the consistency of the popular nearest neighbor estimate $H_n$ of Kozachenko and Leonenko. Without any smoothness condition we show that the estimate is consistent ($E\{|H_n-H(f)|\}\to 0$ as $n\to \infty$) if and only if $\mathbb{E}\{\log (\Vert X\Vert +1)\}<\infty$. Furthermore, if $X$ has compact support, then $H_n\to H(f)$ almost surely.