Comparison of Bayesian predictive methods for model selection

Abstract

The goal of this paper is to compare several widely used Bayesian model selection methods in practical model selection problems, highlight their differences and give recommendations about the preferred approaches. We focus on the variable subset selection for regression and classification and perform several numerical experiments using both simulated and real world data. The results show that the optimization of a utility estimate such as the cross-validation (CV) score is liable to finding overfitted models due to relatively high variance in the utility estimates when the data is scarce. This can also lead to substantial selection induced bias and optimism in the performance evaluation for the selected model. From a predictive viewpoint, best results are obtained by accounting for model uncertainty by forming the full encompassing model, such as the Bayesian model averaging solution over the candidate models. If the encompassing model is too complex, it can be robustly simplified by the projection method, in which the information of the full model is projected onto the submodels. This approach is substantially less prone to overfitting than selection based on CV-score. Overall, the projection method appears to outperform also the maximum a posteriori model and the selection of the most probable variables. The study also demonstrates that the model selection can greatly benefit from using cross-validation outside the searching process both for guiding the model size selection and assessing the predictive performance of the finally selected model.

A few minor changes; added a few sentences, corrected some grammatical errors and modified Figure 7

References in corpus (5)

• Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
• Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory
• Bayesian variable selection with shrinking and diffusing priors
• The Horseshoe Estimator: Posterior Concentration around Nearly Black Vectors
• Projection predictive variable selection using Stan+R

Cited by in corpus (39)

• Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
• Using stacking to average Bayesian predictive distributions
• Sparsity information and regularization in the horseshoe and other shrinkage priors
• Statistical methods for linguistic research: Foundational Ideas - Part II
• Bayesian Item Response Modeling in R with brms and Stan
• A review of predictive uncertainty estimation with machine learning
• Projective Inference in High-dimensional Problems: Prediction and Feature Selection
• Bayesian comparison of latent variable models: Conditional vs marginal likelihoods
• Marginal likelihood computation for model selection and hypothesis testing: an extensive review
• Spatially Resolved Stellar Kinematics of the Ultra-Diffuse Galaxy Dragonfly 44: II. Constraints on Fuzzy Dark Matter
• Bayesian hierarchical stacking: Some models are (somewhere) useful
• Implicitly Adaptive Importance Sampling
• Model interpretation through lower-dimensional posterior summarization
• Projection predictive model selection for Gaussian processes
• Forecasting: theory and practice
• On the safe use of prior densities for Bayesian model selection
• Projection Predictive Inference for Generalized Linear and Additive Multilevel Models
• Limitations of "Limitations of Bayesian leave-one-out cross-validation for model selection"
• Projection predictive variable selection using Stan+R
• Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy
• Fully Bayesian Classification with Heavy-tailed Priors for Selection in High-dimensional Features with Grouping Structure
• lgpr: An interpretable nonparametric method for inferring covariate effects from longitudinal data
• Decision-Making Under Uncertainty in Research Synthesis: Designing for the Garden of Forking Paths
• Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model Comparison
• Dimension-Grouped Mixed Membership Models for Multivariate Categorical Data
• Efficient estimation and correction of selection-induced bias with order statistics
• Bayesian Estimation of Gaussian Graphical Models with Predictive Covariance Selection
• A Parsimonious Tour of Bayesian Model Uncertainty
• Latent space projection predictive inference
• Probabilistically-autoencoded horseshoe-disentangled multidomain item-response theory models
• Projection predictive variable selection for discrete response families with finite support
• Information-based inference for singular models and finite sample sizes: A frequentist information criterion
• Uncertainty Quantification of Graph Convolution Neural Network Models of Evolving Processes
• Model Averaging and its Use in Economics
• Sparse encoding for more-interpretable feature-selecting representations in probabilistic matrix factorization
• Posterior Dispersion Indices
• The ARR2 prior: flexible predictive prior definition for Bayesian auto-regressions
• Marginal likelihood based model comparison in Fuzzy Bayesian Learning
• Disjunct Support Spike and Slab Priors for Variable Selection in Regression under Quasi-sparseness