Valid prediction intervals for regression problems

Abstract

Over the last few decades, various methods have been proposed for estimating prediction intervals in regression settings, including Bayesian methods, ensemble methods, direct interval estimation methods and conformal prediction methods. An important issue is the calibration of these methods: the generated prediction intervals should have a predefined coverage level, without being overly conservative. In this work, we review the above four classes of methods from a conceptual and experimental point of view. Results on benchmark data sets from various domains highlight large fluctuations in performance from one data set to another. These observations can be attributed to the violation of certain assumptions that are inherent to some classes of methods. We illustrate how conformal prediction can be used as a general calibration procedure for methods that deliver poor results without a calibration step.

Minor correction (bibliography and typo in Fig. 3). Thanks to Dr. María Moreno de Castro for spotting this typo

References in corpus (5)

• Variational Inference: A Review for Statisticians
• Recent Advances in Open Set Recognition: A Survey
• Bayesian Neural Networks: An Introduction and Survey
• Regression Conformal Prediction with Nearest Neighbours
• A comparison of some conformal quantile regression methods

Cited by in corpus (7)

• Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons
• How to Evaluate Uncertainty Estimates in Machine Learning for Regression?
• A general framework for multi-step ahead adaptive conformal heteroscedastic time series forecasting
• Lightweight Regression Model with Prediction Interval Estimation for Computer Vision-based Winter Road Surface Condition Monitoring
• Improved conformalized quantile regression
• Uncertainty-Aware Online Extrinsic Calibration: A Conformal Prediction Approach
• Projected random forests and conformal prediction of circular data