Asymptotic bias of the plug-in Shannon entropy estimator under a regularly varying occupancy model

Abstract

Estimating the Shannon entropy of discrete distributions with countably infinite support is a challenging problem. In this paper, we investigate the bias of the plug-in estimator ${\hat{H}}_n$ for the Shannon entropy $H(\mathbf{p})$ under an occupancy model whose frequency sequence exhibits regular variation with tail index $\alpha \in (0,1)$. Using Poissonization and the theory of regular variation, we establish the asymptotic relation $|\mathsf{E}[{\hat{H}}_n]-H(\mathbf{p})|\sim n^{\alpha -1}L(n)C_{\alpha }$, where $L$ is a slowly varying function and $C_{\alpha }$ is an explicit constant depending only on $\alpha$ that admits an integral representation. Our result shows that the asymptotic behavior of the bias of the plug-in estimator under power-law frequency distributions is determined by the tail behavior of the underlying distribution.

7 pages, 1 figure