Bayesian model selection in the $\mathcal{M}$-open setting -- Approximate posterior inference and probability-proportional-to-size subsampling for efficient large-scale leave-one-out cross-validation

Abstract

Comparison of competing statistical models is an essential part of psychological research. From a Bayesian perspective, various approaches to model comparison and selection have been proposed in the literature. However, the applicability of these approaches strongly depends on the assumptions about the model space $\mathcal{M}$, the so-called model view. Furthermore, traditional methods like leave-one-out cross-validation (LOO-CV) estimate the expected log predictive density (ELPD) of a model to investigate how the model generalises out-of-sample, which quickly becomes computationally inefficient when sample size becomes large. Here, we provide a tutorial on approximate Pareto-smoothed importance sampling leave-one-out cross-validation (PSIS-LOO), a computationally efficient method for Bayesian model comparison. First, we discuss several model views and the available Bayesian model comparison methods in each. We then use Bayesian logistic regression as a running example how to apply the method in practice, and show that it outperforms other methods like LOO-CV or information criteria in terms of computational effort while providing similarly accurate ELPD estimates. In a second step, we show how even large-scale models can be compared efficiently by using posterior approximations in combination with probability-proportional-to-size subsampling. We show how to compare competing models based on the ELPD estimates provided, and how to conduct posterior predictive checks to safeguard against overconfidence in one of the models under consideration. We conclude that the method is attractive for mathematical psychologists who aim at comparing several competing statistical models, which are possibly high-dimensional and in the big-data regime.

References in corpus (11)

• Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
• Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory
• Visualization in Bayesian workflow
• A Conceptual Introduction to Hamiltonian Monte Carlo
• Pareto Smoothed Importance Sampling
• Projective Inference in High-dimensional Problems: Prediction and Feature Selection
• Automatic Variational Inference in Stan
• Bayesian leave-one-out cross-validation approximations for Gaussian latent variable models
• Separable states with unique decompositions
• Bayesian leave-one-out cross-validation for large data
• Bayesian projection approaches to variable selection and exploring model uncertainty