Learning Sparse Structured Ensembles with SG-MCMC and Network Pruning

Abstract

An ensemble of neural networks is known to be more robust and accurate than an individual network, however usually with linearly-increased cost in both training and testing. In this work, we propose a two-stage method to learn Sparse Structured Ensembles (SSEs) for neural networks. In the first stage, we run SG-MCMC with group sparse priors to draw an ensemble of samples from the posterior distribution of network parameters. In the second stage, we apply weight-pruning to each sampled network and then perform retraining over the remained connections. In this way of learning SSEs with SG-MCMC and pruning, we not only achieve high prediction accuracy since SG-MCMC enhances exploration of the model-parameter space, but also reduce memory and computation cost significantly in both training and testing of NN ensembles. This is thoroughly evaluated in the experiments of learning SSE ensembles of both FNNs and LSTMs. For example, in LSTM based language modeling (LM), we obtain 21% relative reduction in LM perplexity by learning a SSE of 4 large LSTM models, which has only 30% of model parameters and 70% of computations in total, as compared to the baseline large LSTM LM. To the best of our knowledge, this work represents the first methodology and empirical study of integrating SG-MCMC, group sparse prior and network pruning together for learning NN ensembles.

References in corpus (18)

• Distilling the Knowledge in a Neural Network
• Recurrent Neural Network Regularization
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Stochastic Gradient Hamiltonian Monte Carlo
• Learning Structured Sparsity in Deep Neural Networks
• Regularizing and Optimizing LSTM Language Models
• Bayesian Dark Knowledge
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• Exploring Sparsity in Recurrent Neural Networks
• Learning Intrinsic Sparse Structures within Long Short-Term Memory
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Optimizing Performance of Recurrent Neural Networks on GPUs
• Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
• Group Sparse Priors for Covariance Estimation
• The Relative Performance of Ensemble Methods with Deep Convolutional Neural Networks for Image Classification
• Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling