Consistent Sparse Deep Learning: Theory and Computation

Abstract

Deep learning has been the engine powering many successes of data science. However, the deep neural network (DNN), as the basic model of deep learning, is often excessively over-parameterized, causing many difficulties in training, prediction and interpretation. We propose a frequentist-like method for learning sparse DNNs and justify its consistency under the Bayesian framework: the proposed method could learn a sparse DNN with at most $O(n/\log (n))$ connections and nice theoretical guarantees such as posterior consistency, variable selection consistency and asymptotically optimal generalization bounds. In particular, we establish posterior consistency for the sparse DNN with a mixture Gaussian prior, show that the structure of the sparse DNN can be consistently determined using a Laplace approximation-based marginal posterior inclusion probability approach, and use Bayesian evidence to elicit sparse DNNs learned by an optimization method such as stochastic gradient descent in multiple runs with different initializations. The proposed method is computationally more efficient than standard Bayesian methods for large-scale sparse DNNs. The numerical results indicate that the proposed method can perform very well for large-scale network compression and high-dimensional nonlinear variable selection, both advancing interpretable machine learning.

Accepted by JASA

References in corpus (28)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• On Calibration of Modern Neural Networks
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Weight Uncertainty in Neural Networks
• Spike and slab variable selection: Frequentist and Bayesian strategies
• Random Erasing Data Augmentation
• Predicting Parameters in Deep Learning
• Group Sparse Regularization for Deep Neural Networks
• Bayesian Compression for Deep Learning
• Dropout Training as Adaptive Regularization
• Neural Variational Inference and Learning in Belief Networks
• Learning the Number of Neurons in Deep Networks
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Dynamic Model Pruning with Feedback
• Learning Sparse Networks Using Targeted Dropout
• Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection
• Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classification
• Nearly optimal Bayesian Shrinkage for High Dimensional Regression
• Model Selection in Bayesian Neural Networks via Horseshoe Priors
• Theory of Deep Learning IIb: Optimization Properties of SGD
• Neural networks and rational functions
• Posterior Concentration for Sparse Deep Learning
• Transformed ${\ell }_1$ Regularization for Learning Sparse Deep Neural Networks
• Extended Stochastic Gradient MCMC for Large-Scale Bayesian Variable Selection
• Weak Convergence Rates of Population versus Single-Chain Stochastic Approximation MCMC Algorithms