Combining Ensembles and Data Augmentation can Harm your Calibration

Abstract

Ensemble methods which average over multiple neural network predictions are a simple approach to improve a model's calibration and robustness. Similarly, data augmentation techniques, which encode prior information in the form of invariant feature transformations, are effective for improving calibration and robustness. In this paper, we show a surprising pathology: combining ensembles and data augmentation can harm model calibration. This leads to a trade-off in practice, whereby improved accuracy by combining the two techniques comes at the expense of calibration. On the other hand, selecting only one of the techniques ensures good uncertainty estimates at the expense of accuracy. We investigate this pathology and identify a compounding under-confidence among methods which marginalize over sets of weights and data augmentation techniques which soften labels. Finally, we propose a simple correction, achieving the best of both worlds with significant accuracy and calibration gains over using only ensembles or data augmentation individually. Applying the correction produces new state-of-the art in uncertainty calibration across CIFAR-10, CIFAR-100, and ImageNet.

References in corpus (18)

• Sequence to Sequence Learning with Neural Networks
• Wide Residual Networks
• On Calibration of Modern Neural Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Highway Networks
• Manifold Mixup: Better Representations by Interpolating Hidden States
• Deep Ensembles: A Loss Landscape Perspective
• Measuring Calibration in Deep Learning
• Augmenting Data with Mixup for Sentence Classification: An Empirical Study
• Evaluating model calibration in classification
• BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
• Remix: Rebalanced Mixup
• Uncertainty Quantification and Deep Ensembles
• Calibration tests in multi-class classification: A unifying framework
• Data Interpolating Prediction: Alternative Interpretation of Mixup
• Improved Robustness to Open Set Inputs via Tempered Mixup

Cited by in corpus (1)

• Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning