Uncertainty Quantification and Deep Ensembles

Abstract

Deep Learning methods are known to suffer from calibration issues: they typically produce over-confident estimates. These problems are exacerbated in the low data regime. Although the calibration of probabilistic models is well studied, calibrating extremely over-parametrized models in the low-data regime presents unique challenges. We show that deep-ensembles do not necessarily lead to improved calibration properties. In fact, we show that standard ensembling methods, when used in conjunction with modern techniques such as mixup regularization, can lead to less calibrated models. This text examines the interplay between three of the most simple and commonly used approaches to leverage deep learning when data is scarce: data-augmentation, ensembling, and post-processing calibration methods. Although standard ensembling techniques certainly help boost accuracy, we demonstrate that the calibration of deep ensembles relies on subtle trade-offs. We also find that calibration methods such as temperature scaling need to be slightly tweaked when used with deep-ensembles and, crucially, need to be executed after the averaging process. Our simulations indicate that this simple strategy can halve the Expected Calibration Error (ECE) on a range of benchmark classification problems compared to standard deep-ensembles in the low data regime.

References in corpus (18)

• A Simple Framework for Contrastive Learning of Visual Representations
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• On Calibration of Modern Neural Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Weight Uncertainty in Neural Networks
• Understanding deep learning requires rethinking generalization
• Momentum Contrast for Unsupervised Visual Representation Learning
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Deep Ensembles: A Loss Landscape Perspective
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Entropy and mutual information in models of deep neural networks
• Stochastic Variational Deep Kernel Learning
• Unsupervised Embedding Learning via Invariant and Spreading Instance Feature
• Less Is More: A Comprehensive Framework for the Number of Components of Ensemble Classifiers

Cited by in corpus (10)

• Multi-resolution partial differential equations preserved learning framework for spatiotemporal dynamics
• Revisiting the Calibration of Modern Neural Networks
• i-PI 3.0: a flexible and efficient framework for advanced atomistic simulations
• Combining Ensembles and Data Augmentation can Harm your Calibration
• Should Ensemble Members Be Calibrated?
• On Stein Variational Neural Network Ensembles
• QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learning
• Learnable Uncertainty under Laplace Approximations
• Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning
• Ex uno plures: Splitting One Model into an Ensemble of Subnetworks