Bayesian variable selection for high dimensional generalized linear models: convergence rates of the fitted densities

Abstract

Bayesian variable selection has gained much empirical success recently in a variety of applications when the number $K$ of explanatory variables $(x_1,...,x_K)$ is possibly much larger than the sample size $n$. For generalized linear models, if most of the $x_j$'s have very small effects on the response $y$, we show that it is possible to use Bayesian variable selection to reduce overfitting caused by the curse of dimensionality $K\gg n$. In this approach a suitable prior can be used to choose a few out of the many $x_j$'s to model $y$, so that the posterior will propose probability densities $p$ that are ``often close'' to the true density $p^{\ast }$ in some sense. The closeness can be described by a Hellinger distance between $p$ and $p^{\ast }$ that scales at a power very close to $n^{-1/2}$, which is the ``finite-dimensional rate'' corresponding to a low-dimensional situation. These findings extend some recent work of Jiang [Technical Report 05-02 (2005) Dept. Statistics, Northwestern Univ.] on consistency of Bayesian variable selection for binary classification.

Published in at http://dx.doi.org/10.1214/009053607000000019 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (2)

• From $\epsilon$-entropy to KL-entropy: Analysis of minimum information complexity density estimation
• Best subset selection, persistence in high-dimensional statistical learning and optimization under $l_1$ constraint

Cited by in corpus (1)

• Gibbs posterior for variable selection in high-dimensional classification and data mining