Revisiting the Calibration of Modern Neural Networks

Abstract

Accurate estimation of predictive uncertainty (model calibration) is essential for the safe application of neural networks. Many instances of miscalibration in modern neural networks have been reported, suggesting a trend that newer, more accurate models produce poorly calibrated predictions. Here, we revisit this question for recent state-of-the-art image classification models. We systematically relate model calibration and accuracy, and find that the most recent models, notably those not using convolutions, are among the best calibrated. Trends observed in prior model generations, such as decay of calibration with distribution shift or model size, are less pronounced in recent architectures. We also show that model size and amount of pretraining do not fully explain these differences, suggesting that architecture is a major determinant of calibration properties.

35th Conference on Neural Information Processing Systems (NeurIPS 2021)

References in corpus (9)

• MLP-Mixer: An all-MLP Architecture for Vision
• One weird trick for parallelizing convolutional neural networks
• MetNet: A Neural Weather Model for Precipitation Forecasting
• BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning
• Evaluating model calibration in classification
• Uncertainty Quantification and Deep Ensembles
• Mitigating Bias in Calibration Error Estimation
• Supervised Transfer Learning at Scale for Medical Imaging
• Temporal Probability Calibration

Cited by in corpus (10)

• Exploring the Limits of Out-of-Distribution Detection
• Localizing Objects with Self-Supervised Transformers and no Labels
• Open-Set Recognition: a Good Closed-Set Classifier is All You Need?
• Ensembles of Vision Transformers as a New Paradigm for Automated Classification in Ecology
• Sample Selection Bias in Machine Learning for Healthcare
• Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability
• Enhanced Isotropy Maximization Loss: Seamless and High-Performance Out-of-Distribution Detection Simply Replacing the SoftMax Loss
• Sparse MoEs meet Efficient Ensembles
• ExCon: Explanation-driven Supervised Contrastive Learning for Image Classification
• A Tale Of Two Long Tails