Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning

Abstract

This paper studies the problem of post-hoc calibration of machine learning classifiers. We introduce the following desiderata for uncertainty calibration: (a) accuracy-preserving, (b) data-efficient, and (c) high expressive power. We show that none of the existing methods satisfy all three requirements, and demonstrate how Mix-n-Match calibration strategies (i.e., ensemble and composition) can help achieve remarkably better data-efficiency and expressive power while provably maintaining the classification accuracy of the original classifier. Mix-n-Match strategies are generic in the sense that they can be used to improve the performance of any off-the-shelf calibrator. We also reveal potential issues in standard evaluation practices. Popular approaches (e.g., histogram-based expected calibration error (ECE)) may provide misleading results especially in small-data regime. Therefore, we propose an alternative data-efficient kernel density-based estimator for a reliable evaluation of the calibration performance and prove its asymptotically unbiasedness and consistency. Our approaches outperform state-of-the-art solutions on both the calibration as well as the evaluation tasks in most of the experimental settings. Our codes are available at https://github.com/zhang64-llnl/Mix-n-Match-Calibration.

ICML 2020

Cited by in corpus (18)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Recent Advances and Applications of Deep Learning Methods in Materials Science
• Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibration
• Mitigating Bias in Calibration Error Estimation
• Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification
• Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning
• Distribution-free binary classification: prediction sets, confidence intervals and calibration
• Probabilistic Neighbourhood Component Analysis: Sample Efficient Uncertainty Estimation in Deep Learning
• Deep Probability Estimation
• Improving Uncertainty Calibration of Deep Neural Networks via Truth Discovery and Geometric Optimization
• Top-label calibration and multiclass-to-binary reductions
• Improving Classifier Confidence using Lossy Label-Invariant Transformations
• Meta-Cal: Well-controlled Post-hoc Calibration by Ranking
• Improving Uncertainty-Error Correspondence in Deep Bayesian Medical Image Segmentation
• The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration
• Quantifying Predictive Uncertainty in Medical Image Analysis with Deep Kernel Learning
• CASU2Net: Cascaded Unification Network by a Two-step Early Fusion for Fault Detection in Offshore Wind Turbines
• Class-Similarity Based Label Smoothing for Confidence Calibration