Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration

Abstract

Class probabilities predicted by most multiclass classifiers are uncalibrated, often tending towards over-confidence. With neural networks, calibration can be improved by temperature scaling, a method to learn a single corrective multiplicative factor for inputs to the last softmax layer. On non-neural models the existing methods apply binary calibration in a pairwise or one-vs-rest fashion. We propose a natively multiclass calibration method applicable to classifiers from any model class, derived from Dirichlet distributions and generalising the beta calibration method from binary classification. It is easily implemented with neural nets since it is equivalent to log-transforming the uncalibrated probabilities, followed by one linear layer and softmax. Experiments demonstrate improved probabilistic predictions according to multiple measures (confidence-ECE, classwise-ECE, log-loss, Brier score) across a wide range of datasets and classifiers. Parameters of the learned Dirichlet calibration map provide insights to the biases in the uncalibrated model.

Accepted for presentation at NeurIPS 2019

Cited by in corpus (48)

• Classifier Calibration: A survey on how to assess and improve predicted class probabilities
• Dark Experience for General Continual Learning: a Strong, Simple Baseline
• Unsolved Problems in ML Safety
• Verified Uncertainty Calibration
• Calibrating Deep Neural Networks using Focal Loss
• Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertainty
• Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
• On Model Calibration for Long-Tailed Object Detection and Instance Segmentation
• Learning Noise Transition Matrix from Only Noisy Labels via Total Variation Regularization
• Disentangling Label Distribution for Long-tailed Visual Recognition
• Intra Order-preserving Functions for Calibration of Multi-Class Neural Networks
• A Comparison of Uncertainty Estimation Approaches in Deep Learning Components for Autonomous Vehicle Applications
• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation
• Cal-SFDA: Source-Free Domain-adaptive Semantic Segmentation with Differentiable Expected Calibration Error
• Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning
• Should Ensemble Members Be Calibrated?
• Soft Calibration Objectives for Neural Networks
• Beyond Conformal Predictors: Adaptive Conformal Inference with Confidence Predictors
• Distribution-free uncertainty quantification for classification under label shift
• Approximating Instance-Dependent Noise via Instance-Confidence Embedding
• Temporal Probability Calibration
• Confidence-based Estimators for Predictive Performance in Model Monitoring
• Quantile Regularization: Towards Implicit Calibration of Regression Models
• Multi-Loss Sub-Ensembles for Accurate Classification with Uncertainty Estimation
• Calibrating Deep Neural Network Classifiers on Out-of-Distribution Datasets
• Spatially Varying Label Smoothing: Capturing Uncertainty from Expert Annotations
• Graceful Degradation and Related Fields
• Calibration of Neural Networks using Splines
• A Provably Accurate Randomized Sampling Algorithm for Logistic Regression
• Deep Probability Estimation
• Increasing Trustworthiness of Deep Neural Networks via Accuracy Monitoring
• Top-label calibration and multiclass-to-binary reductions
• On the Usefulness of the Fit-on-the-Test View on Evaluating Calibration of Classifiers
• Newton-Puiseux Analysis for Interpretability and Calibration of Complex-Valued Neural Networks
• Making Heads and Tails of Models with Marginal Calibration for Sparse Tagsets
• Confidence Calibration with Bounded Error Using Transformations
• Meta-Cal: Well-controlled Post-hoc Calibration by Ranking
• Improving Classifier Confidence using Lossy Label-Invariant Transformations
• On Focal Loss for Class-Posterior Probability Estimation: A Theoretical Perspective
• Learning to Cascade: Confidence Calibration for Improving the Accuracy and Computational Cost of Cascade Inference Systems
• Calibration tests beyond classification
• Probabilistic Object Classification using CNN ML-MAP layers
• Class-Similarity Based Label Smoothing for Confidence Calibration
• Identifying and Exploiting Structures for Reliable Deep Learning
• On Deep Neural Network Calibration by Regularization and its Impact on Refinement
• Frustratingly Easy Uncertainty Estimation for Distribution Shift
• MACEst: The reliable and trustworthy Model Agnostic Confidence Estimator
• Calibrating the Full Predictive Class Distribution of 3D Object Detectors for Autonomous Driving