Structured Ensembles: an Approach to Reduce the Memory Footprint of Ensemble Methods

Abstract

In this paper, we propose a novel ensembling technique for deep neural networks, which is able to drastically reduce the required memory compared to alternative approaches. In particular, we propose to extract multiple sub-networks from a single, untrained neural network by solving an end-to-end optimization task combining differentiable scaling over the original architecture, with multiple regularization terms favouring the diversity of the ensemble. Since our proposal aims to detect and extract sub-structures, we call it Structured Ensemble. On a large experimental evaluation, we show that our method can achieve higher or comparable accuracy to competing methods while requiring significantly less storage. In addition, we evaluate our ensembles in terms of predictive calibration and uncertainty, showing they compare favourably with the state-of-the-art. Finally, we draw a link with the continual learning literature, and we propose a modification of our framework to handle continuous streams of tasks with a sub-linear memory cost. We compare with a number of alternative strategies to mitigate catastrophic forgetting, highlighting advantages in terms of average accuracy and memory.

arXiv commentArticle accepted at Neural Networks

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Explaining and Harnessing Adversarial Examples
• On Calibration of Modern Neural Networks
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Pruning Filters for Efficient ConvNets
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Three scenarios for continual learning
• Deep Ensembles: A Loss Landscape Perspective
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• Don't forget, there is more than forgetting: new metrics for Continual Learning
• BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning
• Continual Learning via Neural Pruning
• Bayesian Neural Networks With Maximum Mean Discrepancy Regularization