Hyperparameter Ensembles for Robustness and Uncertainty Quantification

Abstract

Ensembles over neural network weights trained from different random initialization, known as deep ensembles, achieve state-of-the-art accuracy and calibration. The recently introduced batch ensembles provide a drop-in replacement that is more parameter efficient. In this paper, we design ensembles not only over weights, but over hyperparameters to improve the state of the art in both settings. For best performance independent of budget, we propose hyper-deep ensembles, a simple procedure that involves a random search over different hyperparameters, themselves stratified across multiple random initializations. Its strong performance highlights the benefit of combining models with both weight and hyperparameter diversity. We further propose a parameter efficient version, hyper-batch ensembles, which builds on the layer structure of batch ensembles and self-tuning networks. The computational and memory costs of our method are notably lower than typical ensembles. On image classification tasks, with MLP, LeNet, ResNet 20 and Wide ResNet 28-10 architectures, we improve upon both deep and batch ensembles.

arXiv commentAccepted at NeurIPS 2020

References in corpus (24)

• Adam: A Method for Stochastic Optimization
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Practical Bayesian Optimization of Machine Learning Algorithms
• Popular Ensemble Methods: An Empirical Study
• Wide Residual Networks
• Weight Uncertainty in Neural Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Scalable Bayesian Optimization Using Deep Neural Networks
• Structure Discovery in Nonparametric Regression through Compositional Kernel Search
• Deep Ensembles: A Loss Landscape Perspective
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• Measuring Calibration in Deep Learning
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• Stochastic Hyperparameter Optimization through Hypernetworks
• Bayesian Layers: A Module for Neural Network Uncertainty
• Learning the Structure of Deep Sparse Graphical Models
• How Good is the Bayes Posterior in Deep Neural Networks Really?
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• Optimized Generic Feature Learning for Few-shot Classification across Domains
• Adjustable Real-time Style Transfer
• Neural Ensemble Search for Uncertainty Estimation and Dataset Shift

Cited by in corpus (9)

• Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditions
• Deep Ensembles for Low-Data Transfer Learning
• Collaboration of Experts: Achieving 80% Top-1 Accuracy on ImageNet with 100M FLOPs
• Deep Ensembles from a Bayesian Perspective
• No One Representation to Rule Them All: Overlapping Features of Training Methods
• Ex uno plures: Splitting One Model into an Ensemble of Subnetworks
• Combining Different V1 Brain Model Variants to Improve Robustness to Image Corruptions in CNNs
• Combining Diverse Feature Priors
• Towards Comprehensive Monocular Depth Estimation: Multiple Heads Are Better Than One