On Calibration of Mixup Training for Deep Neural Networks

Abstract

Deep Neural Networks (DNN) represent the state of the art in many tasks. However, due to their overparameterization, their generalization capabilities are in doubt and still a field under study. Consequently, DNN can overfit and assign overconfident predictions -- effects that have been shown to affect the calibration of the confidences assigned to unseen data. Data Augmentation (DA) strategies have been proposed to regularize these models, being Mixup one of the most popular due to its ability to improve the accuracy, the uncertainty quantification and the calibration of DNN. In this work however we argue and provide empirical evidence that, due to its fundamentals, Mixup does not necessarily improve calibration. Based on our observations we propose a new loss function that improves the calibration, and also sometimes the accuracy, of DNN trained with this DA technique. Our loss is inspired by Bayes decision theory and introduces a new training framework for designing losses for probabilistic modelling. We provide state-of-the-art accuracy with consistent improvements in calibration performance. Appendix and code are provided here: https://github.com/jmaronas/calibration_MixupDNN_ARCLoss.pytorch.git

arXiv commentTo appear in S+SSPR2020

References in corpus (15)

• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Wide Residual Networks
• On Calibration of Modern Neural Networks
• Understanding deep learning requires rethinking generalization
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Manifold Mixup: Better Representations by Interpolating Hidden States
• Variational Dropout and the Local Reparameterization Trick
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Do ImageNet Classifiers Generalize to ImageNet?
• Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Calibration of Deep Probabilistic Models with Decoupled Bayesian Neural Networks
• On-manifold Adversarial Data Augmentation Improves Uncertainty Calibration