CRUDE: Calibrating Regression Uncertainty Distributions Empirically

Abstract

Calibrated uncertainty estimates in machine learning are crucial to many fields such as autonomous vehicles, medicine, and weather and climate forecasting. While there is extensive literature on uncertainty calibration for classification, the classification findings do not always translate to regression. As a result, modern models for predicting uncertainty in regression settings typically produce uncalibrated and overconfident estimates. To address these gaps, we present a calibration method for regression settings that does not assume a particular uncertainty distribution over the error: Calibrating Regression Uncertainty Distributions Empirically (CRUDE). CRUDE makes the weaker assumption that error distributions have a constant arbitrary shape across the output space, shifted by predicted mean and scaled by predicted standard deviation. We detail a theoretical connection between CRUDE and conformal inference. Across an extensive set of regression tasks, CRUDE demonstrates consistently sharper, better calibrated, and more accurate uncertainty estimates than state-of-the-art techniques.

ICML 2020 Workshop on Uncertainty & Robustness in Deep Learning

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data
• A tutorial on conformal prediction
• GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Measuring Calibration in Deep Learning
• Verified Uncertainty Calibration
• Concepts and Applications of Conformal Prediction in Computational Drug Discovery
• Estimating and Evaluating Regression Predictive Uncertainty in Deep Object Detectors

Cited by in corpus (4)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity
• Short-Term Solar Irradiance Forecasting Using Calibrated Probabilistic Models
• Astronomical Image Quality Prediction based on Environmental and Telescope Operating Conditions