Gibbs posterior for variable selection in high-dimensional classification and data mining

Abstract

In the popular approach of "Bayesian variable selection" (BVS), one uses prior and posterior distributions to select a subset of candidate variables to enter the model. A completely new direction will be considered here to study BVS with a Gibbs posterior originating in statistical mechanics. The Gibbs posterior is constructed from a risk function of practical interest (such as the classification error) and aims at minimizing a risk function without modeling the data probabilistically. This can improve the performance over the usual Bayesian approach, which depends on a probability model which may be misspecified. Conditions will be provided to achieve good risk performance, even in the presence of high dimensionality, when the number of candidate variables "$K$" can be much larger than the sample size "$n$." In addition, we develop a convenient Markov chain Monte Carlo algorithm to implement BVS with the Gibbs posterior.

Published in at http://dx.doi.org/10.1214/07-AOS547 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (4)

• From $\epsilon$-entropy to KL-entropy: Analysis of minimum information complexity density estimation
• Misspecification in infinite-dimensional Bayesian statistics
• Best subset selection, persistence in high-dimensional statistical learning and optimization under $l_1$ constraint
• Bayesian variable selection for high dimensional generalized linear models: convergence rates of the fitted densities

Cited by in corpus (32)

• A General Framework for Updating Belief Distributions
• Calibrating general posterior credible regions
• On the properties of variational approximations of Gibbs posteriors
• Empirical Bayes posterior concentration in sparse high-dimensional linear models
• Ensemble Kalman methods for high-dimensional hierarchical dynamic space-time models
• User-friendly introduction to PAC-Bayes bounds
• Asymptotically minimax empirical Bayes estimation of a sparse normal mean vector
• Quasi-Bayesian analysis of nonparametric instrumental variables models
• Posterior consistency of nonparametric conditional moment restricted models
• Asymptotic normality, concentration, and coverage of generalized posteriors
• Gibbs posterior concentration rates under sub-exponential type losses
• Model-free posterior inference on the area under the receiver operating characteristic curve
• Generalized Geographically Weighted Regression Model within a Modularized Bayesian Framework
• Approximating Bayes in the 21st Century
• Gibbs posterior inference on multivariate quantiles
• Adaptive Variable Selection for Sequential Prediction in Multivariate Dynamic Models
• Empirical Risk Minimization with Relative Entropy Regularization
• On Oracle Property and Asymptotic Validity of Bayesian Generalized Method of Moments
• Robust and rate-optimal Gibbs posterior inference on the boundary of a noisy image
• Bayesian fractional posteriors
• Introducing prior information in Weighted Likelihood Bootstrap with applications to model misspecification
• An approach to large-scale Quasi-Bayesian inference with spike-and-slab priors
• General Robust Bayes Pseudo-Posterior: Exponential Convergence results with Applications
• Semiparametric inference using fractional posteriors
• Comment on Article by Dawid and Musio
• Gaussian Approximation of General Nonparametric Posterior Distributions
• Generalized Bayes Approach to Inverse Problems with Model Misspecification
• Double spike Dirichlet priors for structured weighting
• Generalized Posterior Calibration via Sequential Monte Carlo Sampler
• An Application of Bayesian Variable Selection to Spatial Concurrent Linear Models
• Weighted Particle-Based Optimization for Efficient Generalized Posterior Calibration
• Modified Delayed Acceptance MCMC for Quasi-Bayesian Inference with Linear Moment Conditions