Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model Comparison

Abstract

It is useful to estimate the expected predictive performance of models planned to be used for prediction. We focus on leave-one-out cross-validation (LOO-CV), which has become a popular method for estimating predictive performance of Bayesian models. Given two models, we are interested in comparing the predictive performances and associated uncertainty, which can also be used to compute the probability of one model having better predictive performance than the other model. We study the properties of the Bayesian LOO-CV estimator and the related uncertainty quantification for the predictive performance difference, and analyse when a normal approximation of this uncertainty is well calibrated and whether taking into account higher moments could improve the approximation. We provide new results of the properties both theoretically in the linear regression case and empirically for hierarchical linear, latent linear, and spline models and discuss the challenges. We show that problematic cases include: comparing models with similar predictions, misspecified models, and small data. In these cases, there is a weak connection between the distributions of the LOO-CV estimator and its error. We show that that the problematic skewness of the error distribution for the difference, which occurs when the models make similar predictions, does not fade away when the data size grows to infinity in certain situations. Based on the results, we also provide some practical recommendations for the users of Bayesian LOO-CV for comparing predictive performance of models.

90 pages, 22 figures. Update 2025-10-28: Minor revision, clarifications, new case studies

References in corpus (11)

• Rank-normalization, folding, and localization: An improved $\hat{R}$ for assessing convergence of MCMC
• Assessing and tuning brain decoders: cross-validation, caveats, and guidelines
• Using stacking to average Bayesian predictive distributions
• Cross-validation: what does it estimate and how well does it do it?
• Projective Inference in High-dimensional Problems: Prediction and Feature Selection
• Bayesian comparison of latent variable models: Conditional vs marginal likelihoods
• Approximate leave-future-out cross-validation for Bayesian time series models
• Graphical Test for Discrete Uniformity and its Applications in Goodness of Fit Evaluation and Multiple Sample Comparison
• Bayesian hierarchical stacking: Some models are (somewhere) useful
• Implicitly Adaptive Importance Sampling
• Efficient estimation and correction of selection-induced bias with order statistics