Approximate leave-future-out cross-validation for Bayesian time series models

Abstract

One of the common goals of time series analysis is to use the observed series to inform predictions for future observations. In the absence of any actual new data to predict, cross-validation can be used to estimate a model's future predictive accuracy, for instance, for the purpose of model comparison or selection. Exact cross-validation for Bayesian models is often computationally expensive, but approximate cross-validation methods have been developed, most notably methods for leave-one-out cross-validation (LOO-CV). If the actual prediction task is to predict the future given the past, LOO-CV provides an overly optimistic estimate because the information from future observations is available to influence predictions of the past. To properly account for the time series structure, we can use leave-future-out cross-validation (LFO-CV). Like exact LOO-CV, exact LFO-CV requires refitting the model many times to different subsets of the data. Using Pareto smoothed importance sampling, we propose a method for approximating exact LFO-CV that drastically reduces the computational costs while also providing informative diagnostics about the quality of the approximation.

arXiv comment28 pages, 15 figures, 2 tables

References in corpus (4)

• Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
• Using stacking to average Bayesian predictive distributions
• Efficient leave-one-out cross-validation for Bayesian non-factorized normal and Student-t models
• Optimal mixture weights in multiple importance sampling

Cited by in corpus (11)

• Bayesian hierarchical stacking: Some models are (somewhere) useful
• Cross-validation based adaptive sampling for Gaussian process models
• Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy
• Innovative Approaches in Soil Carbon Sequestration Modelling for Better Prediction with Limited Data
• A guide to state-space modeling of ecological time series
• A Bayesian Dirichlet Auto-Regressive Moving Average Model for Forecasting Lead Times
• Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model Comparison
• Efficient estimation and correction of selection-induced bias with order statistics
• Warped Dynamic Linear Models for Time Series of Counts
• Decomposing Gaussians with Unknown Covariance
• The ARR2 prior: flexible predictive prior definition for Bayesian auto-regressions