Class-Distribution-Aware Calibration for Long-Tailed Visual Recognition
arXiv:2109.05263
Abstract
Despite impressive accuracy, deep neural networks are often miscalibrated and tend to overly confident predictions. Recent techniques like temperature scaling (TS) and label smoothing (LS) show effectiveness in obtaining a well-calibrated model by smoothing logits and hard labels with scalar factors, respectively. However, the use of uniform TS or LS factor may not be optimal for calibrating models trained on a long-tailed dataset where the model produces overly confident probabilities for high-frequency classes. In this study, we propose class-distribution-aware TS (CDA-TS) and LS (CDA-LS) by incorporating class frequency information in model calibration in the context of long-tailed distribution. In CDA-TS, the scalar temperature value is replaced with the CDA temperature vector encoded with class frequency to compensate for the over-confidence. Similarly, CDA-LS uses a vector smoothing factor and flattens the hard labels according to their corresponding class distribution. We also integrate CDA optimal temperature vector with distillation loss, which reduces miscalibration in self-distillation (SD). We empirically show that class-distribution-aware TS and LS can accommodate the imbalanced data distribution yielding superior performance in both calibration error and predictive accuracy. We also observe that SD with an extremely imbalanced dataset is less effective in terms of calibration performance. Code is available in https://github.com/mobarakol/Class-Distribution-Aware-TS-LS.
arXiv commentPresented at the ICML 2021 Workshop on Uncertainty and Robustness in Deep Learning
References in corpus (12)
- Distilling the Knowledge in a Neural Network
- mixup: Beyond Empirical Risk Minimization
- A systematic study of the class imbalance problem in convolutional neural networks
- Understanding deep learning requires rethinking generalization
- When Does Label Smoothing Help?
- Balanced Meta-Softmax for Long-Tailed Visual Recognition
- Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss
- Moonshine: Distilling with Cheap Convolutions
- Calibrating Deep Neural Networks using Focal Loss
- Distillation Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
- Well-calibrated Model Uncertainty with Temperature Scaling for Dropout Variational Inference
- Attended Temperature Scaling: A Practical Approach for Calibrating Deep Neural Networks