Using stacking to average Bayesian predictive distributions

Abstract

The widely recommended procedure of Bayesian model averaging is flawed in the M-open setting in which the true data-generating process is not one of the candidate models being fit. We take the idea of stacking from the point estimation literature and generalize to the combination of predictive distributions, extending the utility function to any proper scoring rule, using Pareto smoothed importance sampling to efficiently compute the required leave-one-out posterior distributions and regularization to get more stability. We compare stacking of predictive distributions to several alternatives: stacking of means, Bayesian model averaging (BMA), pseudo-BMA using AIC-type weighting, and a variant of pseudo-BMA that is stabilized using the Bayesian bootstrap. Based on simulations and real-data applications, we recommend stacking of predictive distributions, with BB-pseudo-BMA as an approximate alternative when computation cost is an issue.

References in corpus (5)

• Variational Inference: A Review for Statisticians
• Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory
• Issues in Stacked Generalization
• Using stacking to average Bayesian predictive distributions
• Automatic Differentiation Variational Inference

Cited by in corpus (43)

• Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions
• Using stacking to average Bayesian predictive distributions
• Relativistic redshift of the star S0-2 orbiting the Galactic center supermassive black hole
• A review of predictive uncertainty estimation with machine learning
• Stacking Models for Nearly Optimal Link Prediction in Complex Networks
• KiDS-1000 Cosmology: constraints beyond flat $\Lambda$CDM
• Projective Inference in High-dimensional Problems: Prediction and Feature Selection
• Bayesian comparison of latent variable models: Conditional vs marginal likelihoods
• Approximate leave-future-out cross-validation for Bayesian time series models
• Boba: Authoring and Visualizing Multiverse Analyses
• Bayesian Model-Averaged Meta-Analysis in Medicine
• Bayesian forecasting of multivariate time series: Scalability, structure uncertainty and decisions
• Towards Precise and Accurate Calculations of Neutrinoless Double-Beta Decay: Project Scoping Workshop Report
• Bayesian hierarchical stacking: Some models are (somewhere) useful
• The Magnificent Five Images of Supernova Refsdal: Time Delay and Magnification Measurements
• A framework for probabilistic weather forecast post-processing across models and lead times using machine learning
• Exploring physics of ferroelectric domain walls via Bayesian analysis of atomically resolved STEM data
• Merging satellite and gauge-measured precipitation using LightGBM with an emphasis on extreme quantiles
• Bayesian Median Autoregression for Robust Time Series Forecasting
• Projecting UK Mortality using Bayesian Generalised Additive Models
• Hot Rocks Survey II: The thermal emission of TOI-1468 b reveals a hot bare rock
• Robust leave-one-out cross-validation for high-dimensional Bayesian models
• Weak evidence for variable occultation depth of 55 Cnc e with TESS
• Holes in Bayesian Statistics
• Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy
• Adaptive Variable Selection for Sequential Prediction in Multivariate Dynamic Models
• Bayesian inference in band excitation Scanning Probe Microscopy for optimal dynamic model selection in imaging
• Assessing Bayes factor surfaces using interactive visualization and computer surrogate modeling
• A Bayesian semi-parametric approach for modeling memory decay in dynamic social networks
• Ensemble learning for blending gridded satellite and gauge-measured precipitation data
• Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model Comparison
• Interpolating between small- and large-$g$ expansions using Bayesian Model Mixing
• Assessing inter-rater reliability with heterogeneous variance components models: Flexible approach accounting for contextual variables
• Does Bayesian Model Averaging improve polynomial extrapolations? Two toy problems as tests
• Observations of scattered light from exoplanet atmospheres
• A loss discounting framework for model averaging and selection in time series models
• A generalized statistical model for fits to parton distributions
• A review of machine learning concepts and methods for addressing challenges in probabilistic hydrological post-processing and forecasting
• Fusion of Gaussian Processes Predictions with Monte Carlo Sampling
• Predicting competitions by combining conditional logistic regression and subjective Bayes: An Academy Awards case study
• Combinations of distributional regression algorithms with application in uncertainty estimation of corrected satellite precipitation products
• Ensemble learning for uncertainty estimation with application to the correction of satellite precipitation products
• Automating Model Comparison in Factor Graphs