Neural Ensemble Search for Uncertainty Estimation and Dataset Shift

Abstract

Ensembles of neural networks achieve superior performance compared to stand-alone networks in terms of accuracy, uncertainty calibration and robustness to dataset shift. \emph{Deep ensembles}, a state-of-the-art method for uncertainty estimation, only ensemble random initializations of a \emph{fixed} architecture. Instead, we propose two methods for automatically constructing ensembles with \emph{varying} architectures, which implicitly trade-off individual architectures' strengths against the ensemble's diversity and exploit architectural variation as a source of diversity. On a variety of classification tasks and modern architecture search spaces, we show that the resulting ensembles outperform deep ensembles not only in terms of accuracy but also uncertainty calibration and robustness to dataset shift. Our further analysis and ablation studies provide evidence of higher ensemble diversity due to architectural variation, resulting in ensembles that can outperform deep ensembles, even when having weaker average base learners. To foster reproducibility, our code is available: \url{https://github.com/automl/nes}

arXiv commentAccepted at NeurIPS 2021; earlier version of this work was accepted for oral presentation at ICML 2020 Workshop on Uncertainty & Robustness in Deep Learning

References in corpus (23)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• End to End Learning for Self-Driving Cars
• Neural Architecture Search: A Survey
• On Calibration of Modern Neural Networks
• DARTS: Differentiable Architecture Search
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• Efficient Architecture Search by Network Transformation
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport
• Understanding and Robustifying Differentiable Architecture Search
• Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Snapshot Ensembles: Train 1, get M for free
• NAS evaluation is frustratingly hard
• When does Diversity Help Generalization in Classification Ensembles?
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
• Bayesian Hyperparameter Optimization for Ensemble Learning
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Sub-Architecture Ensemble Pruning in Neural Architecture Search
• To Ensemble or Not Ensemble: When does End-To-End Training Fail?
• Improving Neural Architecture Search Image Classifiers via Ensemble Learning

Cited by in corpus (7)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Unsolved Problems in ML Safety
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Ex uno plures: Splitting One Model into an Ensemble of Subnetworks
• Towards Comprehensive Monocular Depth Estimation: Multiple Heads Are Better Than One