Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification

Abstract

Among the many ways of quantifying uncertainty in a regression setting, specifying the full quantile function is attractive, as quantiles are amenable to interpretation and evaluation. A model that predicts the true conditional quantiles for each input, at all quantile levels, presents a correct and efficient representation of the underlying uncertainty. To achieve this, many current quantile-based methods focus on optimizing the so-called pinball loss. However, this loss restricts the scope of applicable regression models, limits the ability to target many desirable properties (e.g. calibration, sharpness, centered intervals), and may produce poor conditional quantiles. In this work, we develop new quantile methods that address these shortcomings. In particular, we propose methods that can apply to any class of regression model, allow for selecting a trade-off between calibration and sharpness, optimize for calibration of centered intervals, and produce more accurate conditional quantiles. We provide a thorough experimental evaluation of our methods, which includes a high dimensional uncertainty quantification task in nuclear fusion.

arXiv commentAppears in Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS 2021)

References in corpus (14)

• On Calibration of Modern Neural Networks
• Weight Uncertainty in Neural Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• MOPO: Model-based Offline Policy Optimization
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach
• Single-Model Uncertainties for Deep Learning
• Calibration for the (Computationally-Identifiable) Masses
• Fast Nonparametric Conditional Density Estimation
• Accurate Uncertainty Estimation and Decomposition in Ensemble Learning
• Calibrated Model-Based Deep Reinforcement Learning
• Individual Calibration with Randomized Forecasting
• Reliable training and estimation of variance networks

Cited by in corpus (11)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Classifier Calibration: A survey on how to assess and improve predicted class probabilities
• Regression Diagnostics meets Forecast Evaluation: Conditional Calibration, Reliability Diagrams, and Coefficient of Determination
• Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification
• Prediction uncertainty validation for computational chemists
• MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
• Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity
• Improving Conditional Coverage via Orthogonal Quantile Regression
• Calibrated Multiple-Output Quantile Regression with Representation Learning
• SAUC: Sparsity-Aware Uncertainty Calibration for Spatiotemporal Prediction with Graph Neural Networks
• $f$-Cal: Calibrated aleatoric uncertainty estimation from neural networks for robot perception