Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC

Abstract

Leave-one-out cross-validation (LOO) and the widely applicable information criterion (WAIC) are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model using the log-likelihood evaluated at the posterior simulations of the parameter values. LOO and WAIC have various advantages over simpler estimates of predictive error such as AIC and DIC but are less used in practice because they involve additional computational steps. Here we lay out fast and stable computations for LOO and WAIC that can be performed using existing simulation draws. We introduce an efficient computation of LOO using Pareto-smoothed importance sampling (PSIS), a new procedure for regularizing importance weights. Although WAIC is asymptotically equal to LOO, we demonstrate that PSIS-LOO is more robust in the finite case with weak priors or influential observations. As a byproduct of our calculations, we also obtain approximate standard errors for estimated predictive errors and for comparing of predictive errors between two models. We implement the computations in an R package called 'loo' and demonstrate using models fit with the Bayesian inference package Stan.

References in corpus (6)

• A survey of cross-validation procedures for model selection
• Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory
• Comparison of Bayesian predictive methods for model selection
• Pareto Smoothed Importance Sampling
• Bayesian leave-one-out cross-validation approximations for Gaussian latent variable models
• Case-deletion importance sampling estimators: Central limit theorems and related results

Cited by in corpus (211)

• Visualization in Bayesian workflow
• Inferring change points in the COVID-19 spreading reveals the effectiveness of interventions
• Using stacking to average Bayesian predictive distributions
• Relativistic redshift of the star S0-2 orbiting the Galactic center supermassive black hole
• Sparsity information and regularization in the horseshoe and other shrinkage priors
• Comparison of Bayesian predictive methods for model selection
• Statistical methods for linguistic research: Foundational Ideas - Part II
• Pareto Smoothed Importance Sampling
• A reflective, metal-rich atmosphere for GJ 1214b from its JWST phase curve
• A review of predictive uncertainty estimation with machine learning
• RR Lyrae stars as standard candles in the Gaia Data Release 2 Era
• KiDS-1000 Cosmology: constraints beyond flat $\Lambda$CDM
• A thousand days after the merger: continued X-ray emission from GW170817
• A High Internal Heat Flux and Large Core in a Warm Neptune Exoplanet
• Projective Inference in High-dimensional Problems: Prediction and Feature Selection
• Bayesian comparison of latent variable models: Conditional vs marginal likelihoods
• Approximate leave-future-out cross-validation for Bayesian time series models
• The Upper Edge of the Neptune Desert Is Stable Against Photoevaporation
• Bayesian latent time joint mixed effect models for multicohort longitudinal data
• Boba: Authoring and Visualizing Multiverse Analyses
• Marginal likelihood computation for model selection and hypothesis testing: an extensive review
• Significance Mode Analysis (SigMA) for hierarchical structures. An application to the Sco-Cen OB association
• A resonant sextuplet of sub-Neptunes transiting the bright star HD 110067
• Monitoring the COVID-19 epidemic with nationwide telecommunication data
• A Joint Mass-Radius-Period Distribution of Exoplanets
• Bayesian Model-Averaged Meta-Analysis in Medicine
• Bayesian leave-one-out cross-validation approximations for Gaussian latent variable models
• Spatially Resolved Stellar Kinematics of the Ultra-Diffuse Galaxy Dragonfly 44: II. Constraints on Fuzzy Dark Matter
• CHEOPS Precision Phase Curve of the Super-Earth 55 Cnc e
• Efficient leave-one-out cross-validation for Bayesian non-factorized normal and Student-t models
• Comparison between different methods of model selection in cosmology
• Unbiased and Efficient Log-Likelihood Estimation with Inverse Binomial Sampling
• Propagation Map Reconstruction via Interpolation Assisted Matrix Completion
• A tale of two mergers: constraints on kilonova detection in two short GRBs at z$\sim$0.5
• On the Application of Bayesian Leave-One-Out Cross-Validation to Exoplanet Atmospheric Analysis
• Runtime Task Scheduling using Imitation Learning for Heterogeneous Many-Core Systems
• Informed attribution of flood changes to decadal variation of atmospheric, catchment and river drivers in Upper Austria
• Bayesian hierarchical stacking: Some models are (somewhere) useful
• Implicitly Adaptive Importance Sampling
• Age and market capitalization drive large price variations of cryptocurrencies
• Evidence for mass-dependent peculiar velocities in compact object binaries: Towards better constraints on natal kicks
• First semi-empirical test of the white dwarf mass-radius relationship using a single white dwarf via astrometric microlensing
• Probing Extremal Gravitational-Wave Events with Coarse-Grained Likelihoods
• Measurements of the Hubble Constant with a Two Rung Distance Ladder: Two Out of Three Ain't Bad
• Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors
• Variations in the ${\Sigma }_{\mathrm{SFR}}-{\Sigma }_{\mathrm{mol}}-{\Sigma }_{\star }$ plane across galactic environments in PHANGS galaxies
• The ALMA-ALPAKA survey II. Evolution of turbulence in galaxy disks across cosmic time: difference between cold and warm gas
• Leave-One-Out Cross-Validation for Bayesian Model Comparison in Large Data
• An Observationally-Derived Kick Distribution for Neutron Stars in Binary Systems
• Applying Bayesian Analysis Guidelines to Empirical Software Engineering Data: The Case of Programming Languages and Code Quality
• Bayesian Inference under Cluster Sampling with Probability Proportional to Size
• Practical Hilbert space approximate Bayesian Gaussian processes for probabilistic programming
• Classifying Antimicrobial and Multifunctional Peptides with Bayesian Network Models
• On the safe use of prior densities for Bayesian model selection
• Detecting Multiple Random Changepoints in Bayesian Piecewise Growth Mixture Models
• A Bayesian Time-Varying Effect Model for Behavioral mHealth Data
• Capturing Humans' Mental Models of AI: An Item Response Theory Approach
• Not All Requirements Prioritization Criteria Are Equal at All Times: A Quantitative Analysis
• Hot Rocks Survey II: The thermal emission of TOI-1468 b reveals a hot bare rock
• Investigating the visible phase-curve variability of 55 Cnc e
• Uncertainty Quantification and Propagation in Surrogate-based Bayesian Inference
• A modeler's guide to extreme value software
• Robust leave-one-out cross-validation for high-dimensional Bayesian models
• Weak evidence for variable occultation depth of 55 Cnc e with TESS
• Limitations of "Limitations of Bayesian leave-one-out cross-validation for model selection"
• Spectral information criterion for automatic elbow detection
• Spatial Power Spectra of Dust across the Local Group: No Constraint on Disc Scale Height
• Challenges in the detection of gases in exoplanet atmospheres
• survextrap: a package for flexible and transparent survival extrapolation
• Universal and Automatic Elbow Detection for Learning the Effective Number of Components in Model Selection Problems
• Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy
• Fast, Scalable Approximations to Posterior Distributions in Extended Latent Gaussian Models
• High-dimensional modeling of spatial and spatio-temporal conditional extremes using INLA and Gaussian Markov random fields
• A microlensing search of 700 million VVV light curves
• Evaluating the Evidence for Water World Populations using Mixture Models
• Probabilistic approach to limited-data computed tomography reconstruction
• Fast and Accurate Estimation of Non-Nested Binomial Hierarchical Models Using Variational Inference
• Structure of Working Memory in Children From 3 to 8 Years Old
• A guide to state-space modeling of ecological time series
• Semi-Modular Inference: enhanced learning in multi-modular models by tempering the influence of components
• Towards Causal Analysis of Empirical Software Engineering Data: The Impact of Programming Languages on Coding Competitions
• Multiplicity of stars with planets in the solar neighbourhood
• Bayesian Mixed Effects Models for Zero-inflated Compositions in Microbiome Data Analysis
• BayesSUR: An R package for high-dimensional multivariate Bayesian variable and covariance selection in linear regression
• A Bayesian semi-parametric approach for modeling memory decay in dynamic social networks
• Towards a data-driven model of hadronization using normalizing flows
• Bayesian models for prediction of the set-difference in volleyball
• Analyzing the Effects of COVID-19 Pandemic on the Energy Demand: the Case of Northern Italy
• Bayesian Non-parametric Quantile Process Regression and Estimation of Marginal Quantile Effects
• Photometric Calibrations of M-dwarf Metallicity with Markov Chain Monte Carlo and Bayesian Inference
• Enhanced monitoring of atmospheric methane from space over the Permian basin with hierarchical Bayesian inference
• Modeling Extremal Streamflow using Deep Learning Approximations and a Flexible Spatial Process
• Fast and accurate Bayesian model criticism and conflict diagnostics using R-INLA
• BAMBI: An R package for Fitting Bivariate Angular Mixture Models
• Bayesian multistate modelling of incomplete chronic disease burden data
• JAGS, NIMBLE, Stan: a detailed comparison among Bayesian MCMC software
• Bayes-raking: Bayesian Finite Population Inference with Known Margins
• LOO and WAIC as Model Selection Methods for Polytomous Items
• Quantifying sources of uncertainty in drug discovery predictions with probabilistic models
• Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model Comparison
• Identifying supermassive black hole recoil in elliptical galaxies
• Bayesian Hierarchical Modelling for Tailoring Metric Thresholds
• Modeling Longitudinal Dynamics of Comorbidities
• Assessing inter-rater reliability with heterogeneous variance components models: Flexible approach accounting for contextual variables
• Efficient estimation and correction of selection-induced bias with order statistics
• A general approximation to nested Bayes factors with informed priors
• Bayesian Estimation of Gaussian Graphical Models with Predictive Covariance Selection
• New Development of Bayesian Variable Selection Criteria for Spatial Point Process with Applications
• The Impact of Elicitation and Contrasting Narratives on Engagement, Recall and Attitude Change with News Articles Containing Data Visualization
• Multivariate Bayesian structured variable selection for pharmacogenomic studies
• Observations of scattered light from exoplanet atmospheres
• Bayesian item response models for citizen science ecological data
• Discovery and characterisation of two Neptune-mass planets orbiting HD 212729 with TESS
• A Bayesian approach to uncover local and temporal determinants of heterogeneity in repeated cross-sectional health surveys
• Multilevel network meta-regression for general likelihoods: synthesis of individual and aggregate data with applications to survival analysis
• Bayesian Modal Regression based on Mixture Distributions
• Small Area Estimation of Inequality Measures using Mixtures of Beta
• Asymptotics of cut distributions and robust modular inference using Posterior Bootstrap
• BayesTime: Bayesian Functional Principal Components for Sparse Longitudinal Data
• Formalising Anti-Discrimination Law in Automated Decision Systems
• Finite Population Survey Sampling: An Unapologetic Bayesian Perspective
• Data-Space Validation of High-Dimensional Models by Comparing Sample Quantiles
• The Lindley paradox: The loss of resolution in Bayesian inference
• An empirical study of Linespots: A novel past-fault algorithm
• Integrated population model reveals human and environment driven changes in Baltic ringed seal (Pusa hispida botnica) demography and behavior
• On the impact of heavy meson production spectra on searches for heavy neutral leptons
• Mesospheric nitric oxide model from SCIAMACHY data
• Bayesian Stacked Parametric Survival with Frailty Components and Interval Censored Failure Times
• Projection predictive variable selection for discrete response families with finite support
• The SPDE approach for Gaussian and non-Gaussian fields: 10 years and still running
• Galactic properties that favour star cluster formation: a statistical view
• Stochastic Volatily Models using Hamiltonian Monte Carlo Methods and Stan
• rstap: An R Package for Spatial Temporal Aggregated Predictor Models
• An index of effective number of variables for uncertainty and reliability analysis in model selection problems
• Approximate Cross-Validation for Structured Models
• Hierarchical Bayesian estimation of motor-evoked potential recruitment curves yields accurate and robust estimates
• Horseshoe-based Bayesian nonparametric estimation of effective population size trajectories
• Bayesianize Fuzziness in the Statistical Analysis of Fuzzy Data
• Model-based metrics: Sample-efficient estimates of predictive model subpopulation performance
• Knobs and dials of retrieving JWST transmission spectra. II. Impacts of pipeline-level differences on retrieval posteriors
• Reality-check for Econophysics: Likelihood-based fitting of physics-inspired market models to empirical data
• Generalized Decomposition Priors on R2
• A psychophysical evaluation of techniques for Mooney image generation
• A Matched Survey for the Enigmatic Low Radio Frequency Transient ILT J225347+862146
• Modelling publication bias and p-hacking
• Bayesian inference for partial orders from random linear extensions: power relations from 12th Century Royal Acta
• Design and Structure Dependent Priors for Scale Parameters in Latent Gaussian Models
• Sparse encoding for more-interpretable feature-selecting representations in probabilistic matrix factorization
• Statistical Models for Repeated Categorical Ratings: The R Package rater
• Selecting fitted models under epistemic uncertainty using a stochastic process on quantile functions
• Spatial Temporal Aggregated Predictors to Examine Built Environment Health Effects
• Functional and variables selection in extreme value models for regional flood frequency analysis
• Multi-Block Sparse Functional Principal Components Analysis for Longitudinal Microbiome Multi-Omics Data
• A Bayesian location-scale joint model for time-to-event and multivariate longitudinal data with association based on within-individual variability
• Prediction can be safely used as a proxy for explanation in causally consistent Bayesian generalized linear models
• Bayesian analysis of flexible Heckman selection models using Hamiltonian Monte Carlo
• Bayesian non-asymptotic extreme value models for environmental data
• Unbiased estimator for the variance of the leave-one-out cross-validation estimator for a Bayesian normal model with fixed variance
• Modeling Spatial Heterogeneity in Exposure Buffers and Risk: A Hierarchical Bayesian Approach
• Characterizing quantile-varying covariate effects under the accelerated failure time model
• Semantic Analysis for Automated Evaluation of the Potential Impact of Research Articles
• Region-Referenced Spectral Power Dynamics of EEG Signals: A Hierarchical Modeling Approach
• Variational Bayes latent class approach for EHR-based phenotyping with large real-world data
• Finite mixture representations of zero-and-$N$-inflated distributions for count-compositional data
• A Method to Assess and Argue for Practical Significance in Software Engineering
• Simultaneous estimation of the effective reproducing number and the detection rate of COVID-19
• Hierarchical spline for time series forecasting: An application to Naval ship engine failure rate
• Non-Parametric Bayesian Inference for Partial Orders with Ties from Rank Data observed with Mallows Noise
• Bayesian Hierarchical Bernoulli-Weibull Mixture Model for Extremely Rare Events
• Euclid preparation. L. Calibration of the linear halo bias in $\Lambda (\nu )$CDM cosmologies
• Group Heterogeneity Assessment for Multilevel Models
• Bridging Simulations and Observations: New Insights into Galaxy Formation Simulations via Out-of-Distribution Detection and Bayesian Model Comparison
• Mapping poverty at multiple geographical scales
• Bayesian method for fitting the low-energy constants in chiral perturbation theory
• Generalized Bayes Approach to Inverse Problems with Model Misspecification
• Bayesian Cumulative Probability Models for Continuous and Mixed Outcomes
• Explainable AI for medical imaging: Explaining pneumothorax diagnoses with Bayesian Teaching
• Effects of Antivaccine Tweets on COVID-19 Vaccinations, Cases, and Deaths
• Financial Data Analysis Using Expert Bayesian Framework For Bankruptcy Prediction
• Bulgeless Evolution And the Rise of Discs (BEARD) III. A numerical simulation view of satellites around Milky-Way analogues
• Statistical Inferences and Predictions for Areal Data and Spatial Data Fusion with Hausdorff--Gaussian Processes
• Bayesian analysis of diffusion-driven multi-type epidemic models with application to COVID-19
• Topological models to infer multiphase interstellar medium properties
• The blood currency of suicidal mass shooters: 60 years of U.S. evidence
• Bayesian Analysis of Formula One Race Results: Disentangling Driver Skill and Constructor Advantage
• Regularization and Hierarchical Prior Distributions for Adjustment with Health Care Claims Data: Rethinking Comorbidity Scores
• An Embedded Diachronic Sense Change Model with a Case Study from Ancient Greek
• W-Kernel and Its Principal Space for Frequentist Evaluation of Bayesian Estimators
• Meta-analysis of dichotomous and ordinal tests without a gold standard
• The ARR2 prior: flexible predictive prior definition for Bayesian auto-regressions
• Hidden Markov modelling of spatio-temporal dynamics of measles in 1750-1850 Finland
• Conformal Bayesian Computation
• A numerically stable online implementation and exploration of WAIC through variations of the predictive density, using NIMBLE
• Quantification of empirical determinacy: the impact of likelihood weighting on posterior location and spread in Bayesian meta-analysis estimated with JAGS and INLA
• Choice functions based multi-objective Bayesian optimisation
• Shrinkage estimation for dose-response modeling in phase II trials with multiple schedules
• Unfolding the Network of Peer Grades: A Latent Variable Approach
• SequentialSamplingModels.jl: Simulating and Evaluating Cognitive Models of Response Times in Julia
• Forecasting Fertility with Parametric Mixture Models
• Mediation analysis in longitudinal intervention studies with an ordinal treatment-dependent confounder
• Reconstructing North Korea's Plutonium Production History with Bayesian Inference-Based Reprocessing Waste Analysis
• An Introduction to Inductive Statistical Inference: from Parameter Estimation to Decision-Making
• Bayesian model selection in the $\mathcal{M}$-open setting -- Approximate posterior inference and probability-proportional-to-size subsampling for efficient large-scale leave-one-out cross-validation
• Data-Driven Constraints on Magnetar Population: No Evidence for a Distinct White Dwarf Channel
• Bayesian leave-one-out cross-validation for astrophysical model comparison using gravitational-wave background data
• Bayesian Inference of Nonlinear Malaria Dynamics in Ghana via an Ensemble Markov Chain Monte Carlo Sampler
• Marginal Data Augmentation for Efficient Bayesian Modeling of Counts and Rates with a Demographic Application
• BDSAR: a new package on Bregman divergence for Bayesian simultaneous autoregressive models
• A universal brown dwarf desert formed between planets and stars
• Posterior Dispersion Indices
• Errors-in-variables Modeling of Personalized Treatment-Response Trajectories