Bayesian variable selection for high dimensional generalized linear models: convergence rates of the fitted densities

Abstract

Bayesian variable selection has gained much empirical success recently in a variety of applications when the number $K$ of explanatory variables $(x_1,...,x_K)$ is possibly much larger than the sample size $n$. For generalized linear models, if most of the $x_j$'s have very small effects on the response $y$, we show that it is possible to use Bayesian variable selection to reduce overfitting caused by the curse of dimensionality $K\gg n$. In this approach a suitable prior can be used to choose a few out of the many $x_j$'s to model $y$, so that the posterior will propose probability densities $p$ that are ``often close'' to the true density $p^{\ast }$ in some sense. The closeness can be described by a Hellinger distance between $p$ and $p^{\ast }$ that scales at a power very close to $n^{-1/2}$, which is the ``finite-dimensional rate'' corresponding to a low-dimensional situation. These findings extend some recent work of Jiang [Technical Report 05-02 (2005) Dept. Statistics, Northwestern Univ.] on consistency of Bayesian variable selection for binary classification.

Published in at http://dx.doi.org/10.1214/009053607000000019 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (5)

• High-dimensional graphs and variable selection with the Lasso
• Boosting for high-dimensional linear models
• From $\epsilon$-entropy to KL-entropy: Analysis of minimum information complexity density estimation
• Misspecification in infinite-dimensional Bayesian statistics
• Best subset selection, persistence in high-dimensional statistical learning and optimization under $l_1$ constraint

Cited by in corpus (9)

• Bayesian variable selection with shrinking and diffusing priors
• Gibbs posterior for variable selection in high-dimensional classification and data mining
• Empirical Bayes posterior concentration in sparse high-dimensional linear models
• Posterior contraction in sparse Bayesian factor models for massive covariance matrices
• Posterior consistency in linear models under shrinkage priors
• Adaptive Bayesian density regression for high-dimensional data
• Necessary and Sufficient Conditions for High-Dimensional Posterior Consistency under $g$-Priors
• On Oracle Property and Asymptotic Validity of Bayesian Generalized Method of Moments
• Some Simple Formulas for Posterior Convergence Rates