Accurate Uncertainties for Deep Learning Using Calibrated Regression

Abstract

Methods for reasoning under uncertainty are a key building block of accurate and reliable machine learning systems. Bayesian methods provide a general framework to quantify uncertainty. However, because of model misspecification and the use of approximate inference, Bayesian uncertainty estimates are often inaccurate -- for example, a 90% credible interval may not contain the true outcome 90% of the time. Here, we propose a simple procedure for calibrating any regression algorithm; when applied to Bayesian and probabilistic models, it is guaranteed to produce calibrated uncertainty estimates given enough data. Our procedure is inspired by Platt scaling and extends previous work on classification. We evaluate this approach on Bayesian linear regression, feedforward, and recurrent neural networks, and find that it consistently outputs well-calibrated credible intervals while improving performance on time series forecasting and model-based reinforcement learning tasks.

arXiv commentICML 2018

References in corpus (8)

• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• On Calibration of Modern Neural Networks
• Weight Uncertainty in Neural Networks
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Estimating Uncertainty Online Against an Adversary

Cited by in corpus (108)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data
• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving
• MOPO: Model-based Offline Policy Optimization
• Reliable deep-learning-based phase imaging with uncertainty quantification
• Classifier Calibration: A survey on how to assess and improve predicted class probabilities
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Measuring Calibration in Deep Learning
• Scalable agent alignment via reward modeling: a research direction
• Multivariate Confidence Calibration for Object Detection
• Unsolved Problems in ML Safety
• Verified Uncertainty Calibration
• A Review of Testing Object-Based Environment Perception for Safe Automated Driving
• Parameters Estimation for the Cosmic Microwave Background with Bayesian Neural Networks
• Predicting binding motifs of complex adsorbates using machine learning with a physics-inspired graph representation
• Long-tail learning via logit adjustment
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Regression Diagnostics meets Forecast Evaluation: Conditional Calibration, Reliability Diagrams, and Coefficient of Determination
• Conformal Prediction Under Feedback Covariate Shift for Biomolecular Design
• Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification
• Molecule Identification with Rotational Spectroscopy and Probabilistic Deep Learning
• Calibration of Deep Probabilistic Models with Decoupled Bayesian Neural Networks
• The long road to calibrated prediction uncertainty in computational chemistry
• The Unreasonable Effectiveness of Deep Evidential Regression
• Uncertainty Sets for Image Classifiers using Conformal Prediction
• Dim but not entirely dark: Extracting the Galactic Center Excess' source-count distribution with neural nets
• Entropic Out-of-Distribution Detection: Seamless Detection of Unknown Examples
• Laplace Redux -- Effortless Bayesian Deep Learning
• AI Research Considerations for Human Existential Safety (ARCHES)
• Operational Calibration: Debugging Confidence Errors for DNNs in the Field
• Evaluating Ensemble Post-Processing for Wind Power Forecasts
• Prediction uncertainty validation for computational chemists
• A Comparison of Uncertainty Estimation Approaches in Deep Learning Components for Autonomous Vehicle Applications
• ZELDA: fitting Lyman-alpha line profiles using deep learning
• Countdown Regression: Sharp and Calibrated Survival Predictions
• Calibrated Reliable Regression using Maximum Mean Discrepancy
• Learnable Bernoulli Dropout for Bayesian Deep Learning
• Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity
• Calibrating Uncertainties in Object Localization Task
• Uncertainty-Aware Machine Translation Evaluation
• PAC Confidence Sets for Deep Neural Networks via Calibrated Prediction
• Attended Temperature Scaling: A Practical Approach for Calibrating Deep Neural Networks
• Valid inferential models for prediction in supervised learning problems
• Calibrated Prediction with Covariate Shift via Unsupervised Domain Adaptation
• Probability Calibration for Knowledge Graph Embedding Models
• Survival Function Matching for Calibrated Time-to-Event Predictions
• Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification
• CRUDE: Calibrating Regression Uncertainty Distributions Empirically
• Hyperparameter-Free Out-of-Distribution Detection Using Softmax of Scaled Cosine Similarity
• Selective prediction-set models with coverage guarantees
• Contextual Dropout: An Efficient Sample-Dependent Dropout Module
• Towards calibrated and scalable uncertainty representations for neural networks
• Deep Evidential Regression
• On Calibration and Out-of-domain Generalization
• Calibrated Adaptive Teacher for Domain Adaptive Intelligent Fault Diagnosis
• Truthful AI: Developing and governing AI that does not lie
• A Financial Service Chatbot based on Deep Bidirectional Transformers
• Distribution-free binary classification: prediction sets, confidence intervals and calibration
• Computational catalyst discovery: Active classification through myopic multiscale sampling
• When and How Mixup Improves Calibration
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
• Revisiting the Evaluation of Uncertainty Estimation and Its Application to Explore Model Complexity-Uncertainty Trade-Off
• Bootstrapping Neural Processes
• Towards Dependable Autonomous Systems Based on Bayesian Deep Learning Components
• Universal uncertainty estimation for nuclear detector signals with neural networks and ensemble learning
• Short-Term Solar Irradiance Forecasting Using Calibrated Probabilistic Models
• Exploring Uncertainty in Deep Learning for Construction of Prediction Intervals
• Ranking over Regression for Bayesian Optimization and Molecule Selection
• Simultaneously Reconciled Quantile Forecasting of Hierarchically Related Time Series
• Astronomical Image Quality Prediction based on Environmental and Telescope Operating Conditions
• Deep Conditional Transformation Models
• Quantile Regularization: Towards Implicit Calibration of Regression Models
• Generative Machine Learning for Multivariate Equity Returns
• Heteroscedastic Calibration of Uncertainty Estimators in Deep Learning
• PAC Prediction Sets Under Covariate Shift
• NOMU: Neural Optimization-based Model Uncertainty
• Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability
• When in Doubt: Neural Non-Parametric Uncertainty Quantification for Epidemic Forecasting
• Enhanced Isotropy Maximization Loss: Seamless and High-Performance Out-of-Distribution Detection Simply Replacing the SoftMax Loss
• On the role of Model Uncertainties in Bayesian Optimization
• Estimation of Accurate and Calibrated Uncertainties in Deterministic models
• Why have a Unified Predictive Uncertainty? Disentangling it using Deep Split Ensembles
• Robustness via Cross-Domain Ensembles
• Gaussian Process Uniform Error Bounds with Unknown Hyperparameters for Safety-Critical Applications
• Reliable Uncertainties for Bayesian Neural Networks using Alpha-divergences
• Top-label calibration and multiclass-to-binary reductions
• ALPaCA vs. GP-based Prior Learning: A Comparison between two Bayesian Meta-Learning Algorithms
• Non-Parametric Calibration for Classification
• Uncertainty Intervals for Graph-based Spatio-Temporal Traffic Prediction
• Real-Time Uncertainty Estimation in Computer Vision via Uncertainty-Aware Distribution Distillation
• Improving Calibration and Out-of-Distribution Detection in Medical Image Segmentation with Convolutional Neural Networks
• Meta-learning using privileged information for dynamics
• Uncertainty Propagation in Node Classification
• Re-calibrating Photometric Redshift Probability Distributions Using Feature-space Regression
• Selecting Datasets for Evaluating an Enhanced Deep Learning Framework
• Entropy-based Active Learning of Graph Neural Network Surrogate Models for Materials Properties
• Bayesian Conditional GAN for MRI Brain Image Synthesis
• Temporal mixture ensemble models for intraday volume forecasting in cryptocurrency exchange markets
• Deep Out-of-Distribution Uncertainty Quantification via Weight Entropy Maximization
• Improving Uncertainty Calibration via Prior Augmented Data
• On-Demand Trajectory Predictions for Interaction Aware Highway Driving
• Evidential Turing Processes
• Unsupervised learning of multimodal image registration using domain adaptation with projected Earth Move's discrepancies
• Learning to Predict Error for MRI Reconstruction
• Machine learning pipeline for battery state of health estimation
• n-MeRCI: A new Metric to Evaluate the Correlation Between Predictive Uncertainty and True Error
• Neural Bootstrapper