Calibrating Deep Neural Networks using Focal Loss

Abstract

Miscalibration - a mismatch between a model's confidence and its correctness - of Deep Neural Networks (DNNs) makes their predictions hard to rely on. Ideally, we want networks to be accurate, calibrated and confident. We show that, as opposed to the standard cross-entropy loss, focal loss [Lin et. al., 2017] allows us to learn models that are already very well calibrated. When combined with temperature scaling, whilst preserving accuracy, it yields state-of-the-art calibrated models. We provide a thorough analysis of the factors causing miscalibration, and use the insights we glean from this to justify the empirically excellent performance of focal loss. To facilitate the use of focal loss in practice, we also provide a principled approach to automatically select the hyperparameter involved in the loss function. We perform extensive experiments on a variety of computer vision and NLP datasets, and with a wide variety of network architectures, and show that our approach achieves state-of-the-art calibration without compromising on accuracy in almost all cases. Code is available at https://github.com/torrvision/focal_calibration.

arXiv commentThis paper was accepted at NeurIPS 2020

References in corpus (5)

• On Calibration of Modern Neural Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration
• Evaluating model calibration in classification
• Calibration tests in multi-class classification: A unifying framework

Cited by in corpus (16)

• Mitigating Bias in Calibration Error Estimation
• Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification
• Speedy Performance Estimation for Neural Architecture Search
• On Calibration and Out-of-domain Generalization
• Class-Distribution-Aware Calibration for Long-Tailed Visual Recognition
• Adaptive Label Smoothing
• Soft Calibration Objectives for Neural Networks
• Confidence-based Estimators for Predictive Performance in Model Monitoring
• Domain Adaptation Techniques for Natural and Medical Image Classification
• Calibration of Neural Networks using Splines
• Deep Probability Estimation
• Query-based Hard-Image Retrieval for Object Detection at Test Time
• Diversifying Dialog Generation via Adaptive Label Smoothing
• On Deep Neural Network Calibration by Regularization and its Impact on Refinement
• Class-Similarity Based Label Smoothing for Confidence Calibration
• Identifying and Exploiting Structures for Reliable Deep Learning