Does Dirichlet Prior Smoothing Solve the Shannon Entropy Estimation Problem?

Abstract

The Dirichlet prior is widely used in estimating discrete distributions and functionals of discrete distributions. In terms of Shannon entropy estimation, one approach is to plug-in the Dirichlet prior smoothed distribution into the entropy functional, while the other one is to calculate the Bayes estimator for entropy under the Dirichlet prior for squared error, which is the conditional expectation. We show that in general they do \emph{not} improve over the maximum likelihood estimator, which plugs-in the empirical distribution into the entropy functional. No matter how we tune the parameters in the Dirichlet prior, this approach cannot achieve the minimax rates in entropy estimation, as recently characterized by Jiao, Venkat, Han, and Weissman, and Wu and Yang. The performance of the minimax rate-optimal estimator with $n$ samples is essentially \emph{at least} as good as that of the Dirichlet smoothed entropy estimators with $n\ln n$ samples. We harness the theory of approximation using positive linear operators for analyzing the bias of plug-in estimators for general functionals under arbitrary statistical models, thereby further consolidating the interplay between these two fields, which was thoroughly developed and exploited by Jiao, Venkat, Han, and Weissman. We establish new results in approximation theory, and apply them to analyze the bias of the Dirichlet prior smoothed plug-in entropy estimator. This interplay between bias analysis and approximation theory is of relevance and consequence far beyond the specific problem setting in this paper.

27 pages, 1 figure, published on IEEE Transactions on Information Theory, merged with https://arxiv.org/abs/1406.6959

References in corpus (6)

• Entropy inference and the James-Stein estimator, with application to nonlinear gene association networks
• Minimax Estimation of the $L_1$ Distance
• Chebyshev polynomials, moment matching, and optimal estimation of the unseen
• Minimax Rate-Optimal Estimation of Divergences between Discrete Distributions
• Beyond Maximum Likelihood: from Theory to Practice
• Estimation of KL Divergence: Optimal Minimax Rate

Cited by in corpus (24)

• Inhomogeneous Hypergraph Clustering with Applications
• Multivariate trace estimation in constant quantum depth
• Ensemble Estimation of Information Divergence
• On Estimation of $L_r$-Norms in Gaussian White Noise Models
• Finite-Sample Concentration of the Multinomial in Relative Entropy
• Importance of Small Probability Events in Big Data: Information Measures, Applications, and Challenges
• Distributed Simulation and Distributed Inference
• Polynomial methods in statistical inference: theory and practice
• Adaptive Estimation of Shannon Entropy
• Ensemble Estimation of Generalized Mutual Information with Applications to Genomics
• Multi-sample Estimation of Bacterial Composition Matrix in Metagenomics Data
• Concentration Inequalities for the Empirical Distribution
• Estimation of KL Divergence: Optimal Minimax Rate
• Chernoff-type Concentration of Empirical Probabilities in Relative Entropy
• On estimating the entropy of shallow circuit outputs
• Bias Correction with Jackknife, Bootstrap, and Taylor Series
• Entropy Rate Estimation for Markov Chains with Large State Space
• Time-Efficient Quantum Entropy Estimator via Samplizer
• Statistical Inference in the Differential Privacy Model
• Locally differentially private estimation of nonlinear functionals of discrete distributions
• Modeling Psychotherapy Dialogues with Kernelized Hashcode Representations: A Nonparametric Information-Theoretic Approach
• Estimating the Fundamental Limits is Easier than Achieving the Fundamental Limits
• Minimax Optimal Additive Functional Estimation with Discrete Distribution
• Sharp Concentration Inequalities for the Centered Relative Entropy