Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC

Abstract

Leave-one-out cross-validation (LOO) and the widely applicable information criterion (WAIC) are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model using the log-likelihood evaluated at the posterior simulations of the parameter values. LOO and WAIC have various advantages over simpler estimates of predictive error such as AIC and DIC but are less used in practice because they involve additional computational steps. Here we lay out fast and stable computations for LOO and WAIC that can be performed using existing simulation draws. We introduce an efficient computation of LOO using Pareto-smoothed importance sampling (PSIS), a new procedure for regularizing importance weights. Although WAIC is asymptotically equal to LOO, we demonstrate that PSIS-LOO is more robust in the finite case with weak priors or influential observations. As a byproduct of our calculations, we also obtain approximate standard errors for estimated predictive errors and for comparing of predictive errors between two models. We implement the computations in an R package called 'loo' and demonstrate using models fit with the Bayesian inference package Stan.

References in corpus (4)

• Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory
• Comparison of Bayesian predictive methods for model selection
• Pareto Smoothed Importance Sampling
• Case-deletion importance sampling estimators: Central limit theorems and related results

Cited by in corpus (210)

• Visualization in Bayesian workflow
• Inferring change points in the COVID-19 spreading reveals the effectiveness of interventions
• Using stacking to average Bayesian predictive distributions
• Relativistic redshift of the star S0-2 orbiting the Galactic center supermassive black hole
• Sparsity information and regularization in the horseshoe and other shrinkage priors
• Comparison of Bayesian predictive methods for model selection
• Statistical methods for linguistic research: Foundational Ideas - Part II
• Pareto Smoothed Importance Sampling
• A reflective, metal-rich atmosphere for GJ 1214b from its JWST phase curve
• A review of predictive uncertainty estimation with machine learning
• RR Lyrae stars as standard candles in the Gaia Data Release 2 Era
• KiDS-1000 Cosmology: constraints beyond flat $\Lambda$CDM
• A thousand days after the merger: continued X-ray emission from GW170817
• A High Internal Heat Flux and Large Core in a Warm Neptune Exoplanet
• Projective Inference in High-dimensional Problems: Prediction and Feature Selection
• Bayesian comparison of latent variable models: Conditional vs marginal likelihoods
• Approximate leave-future-out cross-validation for Bayesian time series models
• The Upper Edge of the Neptune Desert Is Stable Against Photoevaporation
• Bayesian latent time joint mixed effect models for multicohort longitudinal data
• Marginal likelihood computation for model selection and hypothesis testing: an extensive review
• Boba: Authoring and Visualizing Multiverse Analyses
• Significance Mode Analysis (SigMA) for hierarchical structures. An application to the Sco-Cen OB association
• A resonant sextuplet of sub-Neptunes transiting the bright star HD 110067
• Monitoring the COVID-19 epidemic with nationwide telecommunication data
• A Joint Mass-Radius-Period Distribution of Exoplanets
• Bayesian Model-Averaged Meta-Analysis in Medicine
• Spatially Resolved Stellar Kinematics of the Ultra-Diffuse Galaxy Dragonfly 44: II. Constraints on Fuzzy Dark Matter
• Comparison between different methods of model selection in cosmology
• CHEOPS Precision Phase Curve of the Super-Earth 55 Cnc e
• Efficient leave-one-out cross-validation for Bayesian non-factorized normal and Student-t models
• Unbiased and Efficient Log-Likelihood Estimation with Inverse Binomial Sampling
• Propagation Map Reconstruction via Interpolation Assisted Matrix Completion
• A tale of two mergers: constraints on kilonova detection in two short GRBs at z$\sim$0.5
• On the Application of Bayesian Leave-One-Out Cross-Validation to Exoplanet Atmospheric Analysis
• Informed attribution of flood changes to decadal variation of atmospheric, catchment and river drivers in Upper Austria
• Runtime Task Scheduling using Imitation Learning for Heterogeneous Many-Core Systems
• Bayesian hierarchical stacking: Some models are (somewhere) useful
• Implicitly Adaptive Importance Sampling
• Age and market capitalization drive large price variations of cryptocurrencies
• Evidence for mass-dependent peculiar velocities in compact object binaries: Towards better constraints on natal kicks
• Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors
• Probing Extremal Gravitational-Wave Events with Coarse-Grained Likelihoods
• Variations in the ${\Sigma }_{\mathrm{SFR}}-{\Sigma }_{\mathrm{mol}}-{\Sigma }_{\star }$ plane across galactic environments in PHANGS galaxies
• Measurements of the Hubble Constant with a Two Rung Distance Ladder: Two Out of Three Ain't Bad
• First semi-empirical test of the white dwarf mass-radius relationship using a single white dwarf via astrometric microlensing
• The ALMA-ALPAKA survey II. Evolution of turbulence in galaxy disks across cosmic time: difference between cold and warm gas
• An Observationally-Derived Kick Distribution for Neutron Stars in Binary Systems
• Leave-One-Out Cross-Validation for Bayesian Model Comparison in Large Data
• Applying Bayesian Analysis Guidelines to Empirical Software Engineering Data: The Case of Programming Languages and Code Quality
• Bayesian Inference under Cluster Sampling with Probability Proportional to Size
• Practical Hilbert space approximate Bayesian Gaussian processes for probabilistic programming
• Detecting Multiple Random Changepoints in Bayesian Piecewise Growth Mixture Models
• On the safe use of prior densities for Bayesian model selection
• Classifying Antimicrobial and Multifunctional Peptides with Bayesian Network Models
• A Bayesian Time-Varying Effect Model for Behavioral mHealth Data
• Capturing Humans' Mental Models of AI: An Item Response Theory Approach
• Uncertainty Quantification and Propagation in Surrogate-based Bayesian Inference
• Not All Requirements Prioritization Criteria Are Equal at All Times: A Quantitative Analysis
• Investigating the visible phase-curve variability of 55 Cnc e
• Hot Rocks Survey II: The thermal emission of TOI-1468 b reveals a hot bare rock
• Robust leave-one-out cross-validation for high-dimensional Bayesian models
• A modeler's guide to extreme value software
• Limitations of "Limitations of Bayesian leave-one-out cross-validation for model selection"
• Weak evidence for variable occultation depth of 55 Cnc e with TESS
• Challenges in the detection of gases in exoplanet atmospheres
• Spatial Power Spectra of Dust across the Local Group: No Constraint on Disc Scale Height
• Spectral information criterion for automatic elbow detection
• survextrap: a package for flexible and transparent survival extrapolation
• Universal and Automatic Elbow Detection for Learning the Effective Number of Components in Model Selection Problems
• Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy
• High-dimensional modeling of spatial and spatio-temporal conditional extremes using INLA and Gaussian Markov random fields
• A microlensing search of 700 million VVV light curves
• Fast, Scalable Approximations to Posterior Distributions in Extended Latent Gaussian Models
• Evaluating the Evidence for Water World Populations using Mixture Models
• Fast and Accurate Estimation of Non-Nested Binomial Hierarchical Models Using Variational Inference
• Structure of Working Memory in Children From 3 to 8 Years Old
• Probabilistic approach to limited-data computed tomography reconstruction
• A guide to state-space modeling of ecological time series
• Semi-Modular Inference: enhanced learning in multi-modular models by tempering the influence of components
• Bayesian Mixed Effects Models for Zero-inflated Compositions in Microbiome Data Analysis
• Multiplicity of stars with planets in the solar neighbourhood
• Towards Causal Analysis of Empirical Software Engineering Data: The Impact of Programming Languages on Coding Competitions
• BayesSUR: An R package for high-dimensional multivariate Bayesian variable and covariance selection in linear regression
• Enhanced monitoring of atmospheric methane from space over the Permian basin with hierarchical Bayesian inference
• Analyzing the Effects of COVID-19 Pandemic on the Energy Demand: the Case of Northern Italy
• Photometric Calibrations of M-dwarf Metallicity with Markov Chain Monte Carlo and Bayesian Inference
• Bayesian models for prediction of the set-difference in volleyball
• A Bayesian semi-parametric approach for modeling memory decay in dynamic social networks
• Towards a data-driven model of hadronization using normalizing flows
• Bayesian Non-parametric Quantile Process Regression and Estimation of Marginal Quantile Effects
• BAMBI: An R package for Fitting Bivariate Angular Mixture Models
• Modeling Extremal Streamflow using Deep Learning Approximations and a Flexible Spatial Process
• Bayesian multistate modelling of incomplete chronic disease burden data
• Fast and accurate Bayesian model criticism and conflict diagnostics using R-INLA
• Bayes-raking: Bayesian Finite Population Inference with Known Margins
• JAGS, NIMBLE, Stan: a detailed comparison among Bayesian MCMC software
• Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model Comparison
• Quantifying sources of uncertainty in drug discovery predictions with probabilistic models
• LOO and WAIC as Model Selection Methods for Polytomous Items
• A general approximation to nested Bayes factors with informed priors
• Bayesian Hierarchical Modelling for Tailoring Metric Thresholds
• Efficient estimation and correction of selection-induced bias with order statistics
• Identifying supermassive black hole recoil in elliptical galaxies
• Modeling Longitudinal Dynamics of Comorbidities
• Assessing inter-rater reliability with heterogeneous variance components models: Flexible approach accounting for contextual variables
• Bayesian Estimation of Gaussian Graphical Models with Predictive Covariance Selection
• New Development of Bayesian Variable Selection Criteria for Spatial Point Process with Applications
• The Impact of Elicitation and Contrasting Narratives on Engagement, Recall and Attitude Change with News Articles Containing Data Visualization
• Multivariate Bayesian structured variable selection for pharmacogenomic studies
• Observations of scattered light from exoplanet atmospheres
• Asymptotics of cut distributions and robust modular inference using Posterior Bootstrap
• Multilevel network meta-regression for general likelihoods: synthesis of individual and aggregate data with applications to survival analysis
• A Bayesian approach to uncover local and temporal determinants of heterogeneity in repeated cross-sectional health surveys
• Small Area Estimation of Inequality Measures using Mixtures of Beta
• Discovery and characterisation of two Neptune-mass planets orbiting HD 212729 with TESS
• Bayesian item response models for citizen science ecological data
• Bayesian Modal Regression based on Mixture Distributions
• Formalising Anti-Discrimination Law in Automated Decision Systems
• BayesTime: Bayesian Functional Principal Components for Sparse Longitudinal Data
• Finite Population Survey Sampling: An Unapologetic Bayesian Perspective
• On the impact of heavy meson production spectra on searches for heavy neutral leptons
• Mesospheric nitric oxide model from SCIAMACHY data
• The Lindley paradox: The loss of resolution in Bayesian inference
• Data-Space Validation of High-Dimensional Models by Comparing Sample Quantiles
• The SPDE approach for Gaussian and non-Gaussian fields: 10 years and still running
• Projection predictive variable selection for discrete response families with finite support
• Integrated population model reveals human and environment driven changes in Baltic ringed seal (Pusa hispida botnica) demography and behavior
• An empirical study of Linespots: A novel past-fault algorithm
• Bayesian Stacked Parametric Survival with Frailty Components and Interval Censored Failure Times
• Horseshoe-based Bayesian nonparametric estimation of effective population size trajectories
• Stochastic Volatily Models using Hamiltonian Monte Carlo Methods and Stan
• Approximate Cross-Validation for Structured Models
• Hierarchical Bayesian estimation of motor-evoked potential recruitment curves yields accurate and robust estimates
• rstap: An R Package for Spatial Temporal Aggregated Predictor Models
• An index of effective number of variables for uncertainty and reliability analysis in model selection problems
• Galactic properties that favour star cluster formation: a statistical view
• Statistical Models for Repeated Categorical Ratings: The R Package rater
• Model-based metrics: Sample-efficient estimates of predictive model subpopulation performance
• Selecting fitted models under epistemic uncertainty using a stochastic process on quantile functions
• Reality-check for Econophysics: Likelihood-based fitting of physics-inspired market models to empirical data
• Bayesian inference for partial orders from random linear extensions: power relations from 12th Century Royal Acta
• Design and Structure Dependent Priors for Scale Parameters in Latent Gaussian Models
• Bayesianize Fuzziness in the Statistical Analysis of Fuzzy Data
• Functional and variables selection in extreme value models for regional flood frequency analysis
• Modelling publication bias and p-hacking
• Knobs and dials of retrieving JWST transmission spectra. II. Impacts of pipeline-level differences on retrieval posteriors
• Generalized Decomposition Priors on R2
• A psychophysical evaluation of techniques for Mooney image generation
• Multi-Block Sparse Functional Principal Components Analysis for Longitudinal Microbiome Multi-Omics Data
• Spatial Temporal Aggregated Predictors to Examine Built Environment Health Effects
• A Matched Survey for the Enigmatic Low Radio Frequency Transient ILT J225347+862146
• Sparse encoding for more-interpretable feature-selecting representations in probabilistic matrix factorization
• Modeling Spatial Heterogeneity in Exposure Buffers and Risk: A Hierarchical Bayesian Approach
• A Bayesian location-scale joint model for time-to-event and multivariate longitudinal data with association based on within-individual variability
• Finite mixture representations of zero-and-$N$-inflated distributions for count-compositional data
• Bridging Simulations and Observations: New Insights into Galaxy Formation Simulations via Out-of-Distribution Detection and Bayesian Model Comparison
• Euclid preparation. L. Calibration of the linear halo bias in $\Lambda (\nu )$CDM cosmologies
• Non-Parametric Bayesian Inference for Partial Orders with Ties from Rank Data observed with Mallows Noise
• Effects of Antivaccine Tweets on COVID-19 Vaccinations, Cases, and Deaths
• Bayesian analysis of flexible Heckman selection models using Hamiltonian Monte Carlo
• Bayesian method for fitting the low-energy constants in chiral perturbation theory
• Mapping poverty at multiple geographical scales
• Variational Bayes latent class approach for EHR-based phenotyping with large real-world data
• Characterizing quantile-varying covariate effects under the accelerated failure time model
• Prediction can be safely used as a proxy for explanation in causally consistent Bayesian generalized linear models
• Generalized Bayes Approach to Inverse Problems with Model Misspecification
• Semantic Analysis for Automated Evaluation of the Potential Impact of Research Articles
• Bayesian Cumulative Probability Models for Continuous and Mixed Outcomes
• Financial Data Analysis Using Expert Bayesian Framework For Bankruptcy Prediction
• Unbiased estimator for the variance of the leave-one-out cross-validation estimator for a Bayesian normal model with fixed variance
• Explainable AI for medical imaging: Explaining pneumothorax diagnoses with Bayesian Teaching
• Bayesian Hierarchical Bernoulli-Weibull Mixture Model for Extremely Rare Events
• Hierarchical spline for time series forecasting: An application to Naval ship engine failure rate
• Region-Referenced Spectral Power Dynamics of EEG Signals: A Hierarchical Modeling Approach
• A Method to Assess and Argue for Practical Significance in Software Engineering
• Group Heterogeneity Assessment for Multilevel Models
• Simultaneous estimation of the effective reproducing number and the detection rate of COVID-19
• Bayesian non-asymptotic extreme value models for environmental data
• Conformal Bayesian Computation
• Topological models to infer multiphase interstellar medium properties
• Meta-analysis of dichotomous and ordinal tests without a gold standard
• Bayesian Analysis of Formula One Race Results: Disentangling Driver Skill and Constructor Advantage
• SequentialSamplingModels.jl: Simulating and Evaluating Cognitive Models of Response Times in Julia
• Regularization and Hierarchical Prior Distributions for Adjustment with Health Care Claims Data: Rethinking Comorbidity Scores
• A universal brown dwarf desert formed between planets and stars
• Marginal Data Augmentation for Efficient Bayesian Modeling of Counts and Rates with a Demographic Application
• Posterior Dispersion Indices
• Statistical Inferences and Predictions for Areal Data and Spatial Data Fusion with Hausdorff--Gaussian Processes
• An Introduction to Inductive Statistical Inference: from Parameter Estimation to Decision-Making
• Bayesian Inference of Nonlinear Malaria Dynamics in Ghana via an Ensemble Markov Chain Monte Carlo Sampler
• Bulgeless Evolution And the Rise of Discs (BEARD) III. A numerical simulation view of satellites around Milky-Way analogues
• Errors-in-variables Modeling of Personalized Treatment-Response Trajectories
• Mediation analysis in longitudinal intervention studies with an ordinal treatment-dependent confounder
• Bayesian analysis of diffusion-driven multi-type epidemic models with application to COVID-19
• Bayesian leave-one-out cross-validation for astrophysical model comparison using gravitational-wave background data
• Reconstructing North Korea's Plutonium Production History with Bayesian Inference-Based Reprocessing Waste Analysis
• Data-Driven Constraints on Magnetar Population: No Evidence for a Distinct White Dwarf Channel
• The blood currency of suicidal mass shooters: 60 years of U.S. evidence
• Bayesian model selection in the $\mathcal{M}$-open setting -- Approximate posterior inference and probability-proportional-to-size subsampling for efficient large-scale leave-one-out cross-validation
• An Embedded Diachronic Sense Change Model with a Case Study from Ancient Greek
• W-Kernel and Its Principal Space for Frequentist Evaluation of Bayesian Estimators
• The ARR2 prior: flexible predictive prior definition for Bayesian auto-regressions
• BDSAR: a new package on Bregman divergence for Bayesian simultaneous autoregressive models
• Hidden Markov modelling of spatio-temporal dynamics of measles in 1750-1850 Finland
• Choice functions based multi-objective Bayesian optimisation
• Quantification of empirical determinacy: the impact of likelihood weighting on posterior location and spread in Bayesian meta-analysis estimated with JAGS and INLA
• Shrinkage estimation for dose-response modeling in phase II trials with multiple schedules
• Forecasting Fertility with Parametric Mixture Models
• A numerically stable online implementation and exploration of WAIC through variations of the predictive density, using NIMBLE
• Unfolding the Network of Peer Grades: A Latent Variable Approach