Ex uno plures: Splitting One Model into an Ensemble of Subnetworks

Abstract

Monte Carlo (MC) dropout is a simple and efficient ensembling method that can improve the accuracy and confidence calibration of high-capacity deep neural network models. However, MC dropout is not as effective as more compute-intensive methods such as deep ensembles. This performance gap can be attributed to the relatively poor quality of individual models in the MC dropout ensemble and their lack of diversity. These issues can in turn be traced back to the coupled training and substantial parameter sharing of the dropout models. Motivated by this perspective, we propose a strategy to compute an ensemble of subnetworks, each corresponding to a non-overlapping dropout mask computed via a pruning strategy and trained independently. We show that the proposed subnetwork ensembling method can perform as well as standard deep ensembles in both accuracy and uncertainty estimates, yet with a computational efficiency similar to MC dropout. Lastly, using several computer vision datasets like CIFAR10/100, CUB200, and Tiny-Imagenet, we experimentally demonstrate that subnetwork ensembling also consistently outperforms recently proposed approaches that efficiently ensemble neural networks.

References in corpus (19)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Wide Residual Networks
• Rethinking the Value of Network Pruning
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Pruning Filters for Efficient ConvNets
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• Deep Ensembles: A Loss Landscape Perspective
• High-Performance Large-Scale Image Recognition Without Normalization
• BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning
• Continual Learning via Neural Pruning
• Confidence Calibration for Convolutional Neural Networks Using Structured Dropout
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification
• Uncertainty Quantification and Deep Ensembles
• Training independent subnetworks for robust prediction
• Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
• On Power Laws in Deep Ensembles
• Diversity inducing Information Bottleneck in Model Ensembles
• DropCluster: A structured dropout for convolutional networks