Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity

Abstract

Reliable uncertainty quantification (UQ) in machine learning (ML) regression tasks is becoming the focus of many studies in materials and chemical science. It is now well understood that average calibration is insufficient, and most studies implement additional methods testing the conditional calibration with respect to uncertainty, i.e. consistency. Consistency is assessed mostly by so-called reliability diagrams. There exists however another way beyond average calibration, which is conditional calibration with respect to input features, i.e. adaptivity. In practice, adaptivity is the main concern of the final users of a ML-UQ method, seeking for the reliability of predictions and uncertainties for any point in features space. This article aims to show that consistency and adaptivity are complementary validation targets, and that a good consistency does not imply a good adaptivity. Adapted validation methods are proposed and illustrated on a representative example.

arXiv admin note: text overlap with arXiv:2303.07170

References in corpus (24)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• On Calibration of Modern Neural Networks
• Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• Fast and Accurate Uncertainty Estimation in Chemical Machine Learning
• High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach
• A critical review of statistical calibration/prediction models handling data inconsistency and model inadequacy
• Metrics for Benchmarking and Uncertainty Quantification: Quality, Applicability, and a Path to Best Practices for Machine Learning in Chemistry
• Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification
• The long road to calibrated prediction uncertainty in computational chemistry
• A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification
• A Survey on Uncertainty Quantification Methods for Deep Learning
• Reliable Neural Networks for Regression Uncertainty Estimation
• Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification
• CRUDE: Calibrating Regression Uncertainty Distributions Empirically
• Individual Calibration with Randomized Forecasting
• The Peril of Popular Deep Learning Uncertainty Estimation Methods
• A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning
• Improving Conditional Coverage via Orthogonal Quantile Regression
• Local Calibration: Metrics and Recalibration
• Validation of uncertainty quantification metrics: a primer based on the consistency and adaptivity concepts
• Confidence curves for UQ validation: probabilistic reference vs. oracle
• Stratification of uncertainties recalibrated by isotonic regression and its impact on calibration error statistics
• Properties of the ENCE and other MAD-based calibration metrics