A Bernstein-Von Mises Theorem for discrete probability distributions

Abstract

We investigate the asymptotic normality of the posterior distribution in the discrete setting, when model dimension increases with sample size. We consider a probability mass function ${\theta }_0$ on $\mathbbm{N}\setminus \{0\}$ and a sequence of truncation levels $(k_n{)}_n$ satisfying $k_n^3\le n{\inf }_{i\le k_n}{\theta }_0(i).$ Let $\hat{\theta }$ denote the maximum likelihood estimate of $({\theta }_0(i){)}_{i\le k_n}$ and let ${\Delta }_n({\theta }_0)$ denote the $k_n$-dimensional vector which $i$-th coordinate is defined by \sqrt{n} (\hatθ_n(i)-θ_0(i)) for $1\le i\le k_n.$ We check that under mild conditions on ${\theta }_0$ and on the sequence of prior probabilities on the $k_n$-dimensional simplices, after centering and rescaling, the variation distance between the posterior distribution recentered around ${\hat{\theta }}_n$ and rescaled by $\sqrt{n}$ and the $k_n$-dimensional Gaussian distribution $\mathcal{N}({\Delta }_n({\theta }_0),I^{-1}({\theta }_0))$ converges in probability to $0.$ This theorem can be used to prove the asymptotic normality of Bayesian estimators of Shannon and Rényi entropies. The proofs are based on concentration inequalities for centered and non-centered Chi-square (Pearson) statistics. The latter allow to establish posterior concentration rates with respect to Fisher distance rather than with respect to the Hellinger distance as it is commonplace in non-parametric Bayesian statistics.

Published in at http://dx.doi.org/10.1214/08-EJS262 the Electronic Journal of Statistics (http://www.i-journals.org/ejs/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (3)

• Convergence rates of posterior distributions for noniid observations
• Posterior convergence rates of Dirichlet mixtures at smooth densities
• Coding on countably infinite alphabets

Cited by in corpus (14)

• Frequentist Consistency of Variational Bayes
• Nonparametric Bernstein-von Mises theorems in Gaussian white noise
• The semiparametric Bernstein-von Mises theorem
• Bernstein von Mises Theorems for Gaussian Regression with increasing number of regressors
• A Bernstein-von Mises theorem for smooth functionals in semiparametric models
• Adaptive Bernstein-von Mises theorems in Gaussian white noise
• Quasi-Bayesian analysis of nonparametric instrumental variables models
• Finite Sample Bernstein -- von Mises Theorem for Semiparametric Problems
• Controlling Uncertainty of Empirical First-Passage Times in the Small-Sample Regime
• Semiparametric posterior limits under local asymptotic exponentiality
• Non Parametric Hidden Markov Models with Finite State Space: Posterior Concentration Rates
• On frequentist coverage errors of Bayesian credible sets in moderately high dimensions
• The semiparametric Bernstein-von Mises theorem for models with symmetric error
• Semiparametric posterior limits