Neural Ensemble Search for Uncertainty Estimation and Dataset Shift

Abstract

Ensembles of neural networks achieve superior performance compared to stand-alone networks in terms of accuracy, uncertainty calibration and robustness to dataset shift. \emph{Deep ensembles}, a state-of-the-art method for uncertainty estimation, only ensemble random initializations of a \emph{fixed} architecture. Instead, we propose two methods for automatically constructing ensembles with \emph{varying} architectures, which implicitly trade-off individual architectures' strengths against the ensemble's diversity and exploit architectural variation as a source of diversity. On a variety of classification tasks and modern architecture search spaces, we show that the resulting ensembles outperform deep ensembles not only in terms of accuracy but also uncertainty calibration and robustness to dataset shift. Our further analysis and ablation studies provide evidence of higher ensemble diversity due to architectural variation, resulting in ensembles that can outperform deep ensembles, even when having weaker average base learners. To foster reproducibility, our code is available: \url{https://github.com/automl/nes}

Accepted at NeurIPS 2021; earlier version of this work was accepted for oral presentation at ICML 2020 Workshop on Uncertainty & Robustness in Deep Learning

References in corpus (10)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• On Calibration of Modern Neural Networks
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• Efficient Architecture Search by Network Transformation
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• NAS evaluation is frustratingly hard
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Improving Neural Architecture Search Image Classifiers via Ensemble Learning

Cited by in corpus (7)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Unsolved Problems in ML Safety
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Ex uno plures: Splitting One Model into an Ensemble of Subnetworks
• Towards Comprehensive Monocular Depth Estimation: Multiple Heads Are Better Than One