High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach

Abstract

This paper considers the generation of prediction intervals (PIs) by neural networks for quantifying uncertainty in regression tasks. It is axiomatic that high-quality PIs should be as narrow as possible, whilst capturing a specified portion of data. We derive a loss function directly from this axiom that requires no distributional assumption. We show how its form derives from a likelihood principle, that it can be used with gradient descent, and that model uncertainty is accounted for in ensembled form. Benchmark experiments show the method outperforms current state-of-the-art uncertainty quantification methods, reducing average PI width by over 10%.

References in corpus (5)

• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Weight Uncertainty in Neural Networks
• A tutorial on conformal prediction
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks

Cited by in corpus (37)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• How to Evaluate Uncertainty Estimates in Machine Learning for Regression?
• Conformalized Quantile Regression
• Optimal Adaptive Prediction Intervals for Electricity Load Forecasting in Distribution Systems via Reinforcement Learning
• Uncertainty Sets for Image Classifiers using Conformal Prediction
• DeepGLEAM: A hybrid mechanistic and deep learning model for COVID-19 forecasting
• Calibrated Reliable Regression using Maximum Mean Discrepancy
• Automatic Large Scale Detection of Red Palm Weevil Infestation using Aerial and Street View Images
• Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity
• Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification
• Reliable training and estimation of variance networks
• PIVEN: A Deep Neural Network for Prediction Intervals with Specific Value Prediction
• A Simple Probabilistic Method for Deep Classification under Input-Dependent Label Noise
• Quantifying Deep Learning Model Uncertainty in Conformal Prediction
• Uncertainty Quantification in CNN Through the Bootstrap of Convex Neural Networks
• Integrating Uncertainty into Neural Network-based Speech Enhancement
• Prediction Intervals: Split Normal Mixture from Quality-Driven Deep Ensembles
• Uncertainty-Aware Lookahead Factor Models for Quantitative Investing
• Quantifying Intrinsic Uncertainty in Classification via Deep Dirichlet Mixture Networks
• Exploring Uncertainty in Deep Learning for Construction of Prediction Intervals
• PI3NN: Out-of-distribution-aware prediction intervals from three neural networks
• NOMU: Neural Optimization-based Model Uncertainty
• AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantification
• Optimized conformal classification using gradient descent approximation
• Prediction intervals for Deep Neural Networks
• The Monte Carlo Transformer: a stochastic self-attention model for sequence prediction
• Peaking into the Black-box: Prediction Intervals Give Insight into Data-driven Quadrotor Model Reliability
• Approximation to Object Conditional Validity with Inductive Conformal Predictors
• Reliability Quantification of Deep Reinforcement Learning-based Control
• RBUE: A ReLU-Based Uncertainty Estimation Method of Deep Neural Networks
• Probabilistic modeling of lake surface water temperature using a Bayesian spatio-temporal graph convolutional neural network
• Temporal mixture ensemble models for intraday volume forecasting in cryptocurrency exchange markets
• Overestimation learning with guarantees
• HDI-Forest: Highest Density Interval Regression Forest
• Deep Classifiers with Label Noise Modeling and Distance Awareness
• Optimal Prediction Intervals for Macroeconomic Time Series Using Chaos and NSGA II
• Uncertainty Surrogates for Deep Learning