BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning

Abstract

Ensembles, where multiple neural networks are trained individually and their predictions are averaged, have been shown to be widely successful for improving both the accuracy and predictive uncertainty of single neural networks. However, an ensemble's cost for both training and testing increases linearly with the number of networks, which quickly becomes untenable. In this paper, we propose BatchEnsemble, an ensemble method whose computational and memory costs are significantly lower than typical ensembles. BatchEnsemble achieves this by defining each weight matrix to be the Hadamard product of a shared weight among all ensemble members and a rank-one matrix per member. Unlike ensembles, BatchEnsemble is not only parallelizable across devices, where one device trains one member, but also parallelizable within a device, where multiple ensemble members are updated simultaneously for a given mini-batch. Across CIFAR-10, CIFAR-100, WMT14 EN-DE/EN-FR translation, and out-of-distribution tasks, BatchEnsemble yields competitive accuracy and uncertainties as typical ensembles; the speedup at test time is 3X and memory reduction is 3X at an ensemble of size 4. We also apply BatchEnsemble to lifelong learning, where on Split-CIFAR-100, BatchEnsemble yields comparable performance to progressive neural networks while having a much lower computational and memory costs. We further show that BatchEnsemble can easily scale up to lifelong learning on Split-ImageNet which involves 100 sequential learning tasks.

References in corpus (5)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• Popular Ensemble Methods: An Empirical Study
• SGDR: Stochastic Gradient Descent with Warm Restarts

Cited by in corpus (38)

• Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
• Revisiting the Calibration of Modern Neural Networks
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• Training independent subnetworks for robust prediction
• Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models
• Deep Learning Through the Lens of Example Difficulty
• Regression Prior Networks
• Improving the performance of bagging ensembles for data streams through mini-batching
• Diversity Matters When Learning From Ensembles
• Encoders and Ensembles for Task-Free Continual Learning
• Deep Ensembles for Low-Data Transfer Learning
• Graph Posterior Network: Bayesian Predictive Uncertainty for Node Classification
• Combining Ensembles and Data Augmentation can Harm your Calibration
• Multi-task neural networks by learned contextual inputs
• Closer Look at the Uncertainty Estimation in Semantic Segmentation under Distributional Shift
• Robust Semantic Segmentation with Superpixel-Mix
• Diversity and Generalization in Neural Network Ensembles
• Unifying Instance and Panoptic Segmentation with Dynamic Rank-1 Convolutions
• Beyond Point Estimate: Inferring Ensemble Prediction Variation from Neuron Activation Strength in Recommender Systems
• SLURP: Side Learning Uncertainty for Regression Problems
• Bayesian OOD detection with aleatoric uncertainty and outlier exposure
• Continuous-Time Model-Based Reinforcement Learning
• Sparse Uncertainty Representation in Deep Learning with Inducing Weights
• Ex uno plures: Splitting One Model into an Ensemble of Subnetworks
• Robustness via Cross-Domain Ensembles
• Sparse MoEs meet Efficient Ensembles
• Model Agnostic Combination for Ensemble Learning
• Reliability Quantification of Deep Reinforcement Learning-based Control
• Deep Ensemble Collaborative Learning by using Knowledge-transfer Graph for Fine-grained Object Classification
• Neural Parameter Allocation Search
• DEBOSH: Deep Bayesian Shape Optimization
• Structured Ensembles: an Approach to Reduce the Memory Footprint of Ensemble Methods
• Lifelong Mixture of Variational Autoencoders
• Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model
• Greedy Bayesian Posterior Approximation with Deep Ensembles
• Uncertainty Surrogates for Deep Learning
• Combining Different V1 Brain Model Variants to Improve Robustness to Image Corruptions in CNNs