A review of predictive uncertainty estimation with machine learning

Abstract

Predictions and forecasts of machine learning models should take the form of probability distributions, aiming to increase the quantity of information communicated to end users. Although applications of probabilistic prediction and forecasting with machine learning models in academia and industry are becoming more frequent, related concepts and methods have not been formalized and structured under a holistic view of the entire field. Here, we review the topic of predictive uncertainty estimation with machine learning algorithms, as well as the related metrics (consistent scoring functions and proper scoring rules) for assessing probabilistic predictions. The review covers a time period spanning from the introduction of early statistical (linear regression and time series models, based on Bayesian statistics or quantile regression) to recent machine learning algorithms (including generalized additive models for location, scale and shape, random forests, boosting and deep learning algorithms) that are more flexible by nature. The review of the progress in the field, expedites our understanding on how to develop new algorithms tailored to users' needs, since the latest advancements are based on some fundamental concepts applied to more complex algorithms. We conclude by classifying the material and discussing challenges that are becoming a hot topic of research.

arXiv comment89 pages, 5 figures

References in corpus (56)

• XGBoost: A Scalable Tree Boosting System
• Deep Learning in Neural Networks: An Overview
• Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• To Explain or to Predict?
• Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods
• BART: Bayesian additive regression trees
• Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions
• Neural networks for post-processing ensemble weather forecasts
• Using stacking to average Bayesian predictive distributions
• Extremal Quantile Regression: An Overview
• Quantile and Probability Curves Without Crossing
• The Evolution of Boosting Algorithms - From Machine Learning to Statistical Modelling
• Comparison of Bayesian predictive methods for model selection
• Bayesian model averaging: A systematic review and conceptual classification
• Multivariate quantiles and multiple-output regression quantiles: From $L_1$ optimization to halfspace depth
• Extremal quantile regression
• Statistical Postprocessing for Weather Forecasts -- Review, Challenges and Avenues in a Big Data World
• Practical heteroskedastic Gaussian process modeling for large simulation experiments
• Probabilistic Mid- and Long-Term Electricity Price Forecasting
• Regression Conformal Prediction with Nearest Neighbours
• Reliability, Sufficiency, and the Decomposition of Proper Scores
• Forecast verification for extreme value distributions with an application to probabilistic peak wind prediction
• Theory and Applications of Proper Scoring Rules
• Most Likely Transformations
• Conditional Transformation Models
• Probabilistic Forecasting of Sensory Data with Generative Adversarial Networks - ForGAN
• Boosting algorithms in energy research: A systematic review
• Probabilistic wind speed forecasting using Bayesian model averaging with truncated normal components
• A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applications
• Proper local scoring rules
• Stability selection for component-wise gradient boosting in multiple dimensions
• Distributional Regression Forests for Probabilistic Precipitation Forecasting in Complex Terrain
• A comparison of some conformal quantile regression methods
• Extending Statistical Boosting - An Overview of Recent Methodological Developments
• Prediction based on conditional distributions of vine copulas
• Isotonic Distributional Regression
• Probabilistic Gradient Boosting Machines for Large-Scale Probabilistic Regression
• Practical Bayesian Modeling and Inference for Massive Spatial Datasets On Modest Computing Environments
• Bayesian Inference Methods for Univariate and Multivariate GARCH Models: a Survey
• Point forecasting and forecast evaluation with generalized Huber loss
• bamlss: A Lego Toolbox for Flexible Bayesian Regression (and Beyond)
• Time series forecasting with Gaussian Processes needs priors
• Local proper scoring rules of order two
• Scoring Interval Forecasts: Equal-Tailed, Shortest, and Modal Interval
• Why scoring functions cannot assess tail properties
• CRPS Learning
• Forecast Evaluation of Quantiles, Prediction Intervals, and other Set-Valued Functionals
• Bayesian Inference for Regression Copulas
• Contrast Trees and Distribution Boosting
• Evaluating probability forecasts
• M5 Competition Uncertainty: Overdispersion, distributional forecasting, GAMLSS and beyond
• Quantile Regression Neural Networks: A Bayesian Approach
• Bayesian Non-parametric Quantile Process Regression and Estimation of Marginal Quantile Effects
• Marginally-calibrated deep distributional regression
• Properization: Constructing Proper Scoring Rules via Bayes Acts

Cited by in corpus (7)

• Uncertainty estimation of machine learning spatial precipitation predictions from satellite data
• Superior Scoring Rules for Probabilistic Evaluation of Single-Label Multi-Class Classification Tasks
• Deep Huber quantile regression networks
• Combinations of distributional regression algorithms with application in uncertainty estimation of corrected satellite precipitation products
• Ensemble learning for uncertainty estimation with application to the correction of satellite precipitation products
• Personalised dynamic super learning: an application in predicting hemodiafiltration convection volumes
• Multivariate Uncertainty Quantification with Tomographic Quantile Forests