Deep Ensembles: A Loss Landscape Perspective

Abstract

Deep ensembles have been empirically shown to be a promising approach for improving accuracy, uncertainty and out-of-distribution robustness of deep learning models. While deep ensembles were theoretically motivated by the bootstrap, non-bootstrap ensembles trained with just random initialization also perform well in practice, which suggests that there could be other explanations for why deep ensembles work well. Bayesian neural networks, which learn distributions over the parameters of the network, are theoretically well-motivated by Bayesian principles, but do not perform as well as deep ensembles in practice, particularly under dataset shift. One possible explanation for this gap between theory and practice is that popular scalable variational Bayesian methods tend to focus on a single mode, whereas deep ensembles tend to explore diverse modes in function space. We investigate this hypothesis by building on recent work on understanding the loss landscape of neural networks and adding our own exploration to measure the similarity of functions in the space of predictions. Our results show that random initializations explore entirely different modes, while functions along an optimization trajectory or sampled from the subspace thereof cluster within a single mode predictions-wise, while often deviating significantly in the weight space. Developing the concept of the diversity--accuracy plane, we show that the decorrelation power of random initializations is unmatched by popular subspace sampling methods. Finally, we evaluate the relative effects of ensembling, subspace based methods and ensembles of subspace based methods, and the experimental results validate our hypothesis.

References in corpus (8)

• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Qualitatively characterizing neural network optimization problems
• Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• Essentially No Barriers in Neural Network Energy Landscape
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning

Cited by in corpus (96)

• Recent Advances and Applications of Deep Learning Methods in Materials Science
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• Predicting materials properties without crystal structure: Deep representation learning from stoichiometry
• Artificial Intelligence for Digital and Computational Pathology
• Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Revisiting Deep Learning Models for Tabular Data
• What Are Bayesian Neural Network Posteriors Really Like?
• Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditions
• Reinforcement Learning through Active Inference
• Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization
• Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challenge
• Scalable Uncertainty Quantification for Deep Operator Networks using Randomized Priors
• On the Expressiveness of Approximate Inference in Bayesian Neural Networks
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• Deep Ensemble Analysis for Imaging X-ray Polarimetry
• Uncertainty Estimation in Autoregressive Structured Prediction
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification
• Uncertainty Quantification and Deep Ensembles
• Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learning
• Zero Time Waste: Recycling Predictions in Early Exit Neural Networks
• Training independent subnetworks for robust prediction
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Class Balancing Diversity Multimodal Ensemble for Alzheimer's Disease Diagnosis and Early Detection
• Efficient Bayesian Uncertainty Estimation for nnU-Net
• Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel
• Data-Driven Selection and Spectral Classification of White Dwarf Stars
• Connecting Low-Loss Subspace for Personalized Federated Learning
• Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models
• A Comparison of Uncertainty Estimation Approaches in Deep Learning Components for Autonomous Vehicle Applications
• Assessing Generalization of SGD via Disagreement
• Uncertainty Estimation for Heatmap-based Landmark Localization
• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation
• Regression Prior Networks
• Diversity Matters When Learning From Ensembles
• On Power Laws in Deep Ensembles
• Learning under Model Misspecification: Applications to Variational and Ensemble methods
• Repulsive Deep Ensembles are Bayesian
• Learning Neural Network Subspaces
• Distributional Generalization: A New Kind of Generalization
• Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
• Linear Mode Connectivity in Multitask and Continual Learning
• Bayesian Deep Learning via Subnetwork Inference
• Diversity inducing Information Bottleneck in Model Ensembles
• One Versus all for deep Neural Network Incertitude (OVNNI) quantification
• Integrating Uncertainty into Neural Network-based Speech Enhancement
• Combining Ensembles and Data Augmentation can Harm your Calibration
• Leveraging Uncertainty for Improved Static Malware Detection Under Extreme False Positive Constraints
• Deep Ensembles for Low-Data Transfer Learning
• Influence of uncertainty estimation techniques on false-positive reduction in liver lesion detection
• Prediction Intervals: Split Normal Mixture from Quality-Driven Deep Ensembles
• LiBRe: A Practical Bayesian Approach to Adversarial Detection
• Deep Ensembles from a Bayesian Perspective
• Nondeterminism and Instability in Neural Network Optimization
• A Scalable and Transferable Time Series Prediction Framework for Demand Forecasting
• Fast and reliable uncertainty quantification with neural network ensembles for industrial image classification
• Robust Semantic Segmentation with Superpixel-Mix
• Exploiting Uncertainties from Ensemble Learners to Improve Decision-Making in Healthcare AI
• Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles
• On Stein Variational Neural Network Ensembles
• Exploiting Adam-like Optimization Algorithms to Improve the Performance of Convolutional Neural Networks
• No One Representation to Rule Them All: Overlapping Features of Training Methods
• NOMU: Neural Optimization-based Model Uncertainty
• A benchmark study on reliable molecular supervised learning via Bayesian learning
• Efficient Ensemble Model Generation for Uncertainty Estimation with Bayesian Approximation in Segmentation
• Bayesian posterior approximation with stochastic ensembles
• Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling
• Diversity and Generalization in Neural Network Ensembles
• On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay
• Locally Adaptive Label Smoothing for Predictive Churn
• Multi-head Ensemble of Smoothed Classifiers for Certified Robustness
• Dual policy as self-model for planning
• Density Uncertainty Quantification with NeRF-Ensembles: Impact of Data and Scene Constraints
• Uncertainties of Satellite-based Essential Climate Variables from Deep Learning
• Ex uno plures: Splitting One Model into an Ensemble of Subnetworks
• Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning
• Hypothesis Disparity Regularized Mutual Information Maximization
• A Novel Regression Loss for Non-Parametric Uncertainty Optimization
• An information-based metric for observing strategy optimization, demonstrated in the context of photometric redshifts with applications to cosmology
• Quantifying Uncertainty in Deep Spatiotemporal Forecasting
• Continuous Control With Ensemble Deep Deterministic Policy Gradients
• On Efficient Uncertainty Estimation for Resource-Constrained Mobile Applications
• Explaining generalization in deep learning: progress and fundamental limits
• Accurate and Reliable Forecasting using Stochastic Differential Equations
• RBUE: A ReLU-Based Uncertainty Estimation Method of Deep Neural Networks
• Quantifying Epistemic Uncertainty in Deep Learning
• Combining Different V1 Brain Model Variants to Improve Robustness to Image Corruptions in CNNs
• Greedy Bayesian Posterior Approximation with Deep Ensembles
• What can linear interpolation of neural network loss landscapes tell us?
• Leveraging Uncertainty from Deep Learning for Trustworthy Materials Discovery Workflows
• Structured Ensembles: an Approach to Reduce the Memory Footprint of Ensemble Methods
• Neural Bootstrapper
• Probabilistic Spatial Analysis in Quantitative Microscopy with Uncertainty-Aware Cell Detection using Deep Bayesian Regression of Density Maps
• Multi-headed Neural Ensemble Search
• A deep ensemble approach to X-ray polarimetry
• Probabilistic Trust Intervals for Out of Distribution Detection