Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details

Abstract

Sparse deep neural networks have proven to be efficient for predictive model building in large-scale studies. Although several works have studied theoretical and numerical properties of sparse neural architectures, they have primarily focused on the edge selection. Sparsity through edge selection might be intuitively appealing; however, it does not necessarily reduce the structural complexity of a network. Instead pruning excessive nodes leads to a structurally sparse network with significant computational speedup during inference. To this end, we propose a Bayesian sparse solution using spike-and-slab Gaussian priors to allow for automatic node selection during training. The use of spike-and-slab prior alleviates the need of an ad-hoc thresholding rule for pruning. In addition, we adopt a variational Bayes approach to circumvent the computational challenges of traditional Markov Chain Monte Carlo (MCMC) implementation. In the context of node selection, we establish the fundamental result of variational posterior consistency together with the characterization of prior parameters. In contrast to the previous works, our theoretical development relaxes the assumptions of the equal number of nodes and uniform bounds on all network weights, thereby accommodating sparse networks with layer-dependent node structures or coefficient bounds. With a layer-wise characterization of prior inclusion probabilities, we discuss the optimal contraction rates of the variational posterior. We empirically demonstrate that our proposed approach outperforms the edge selection method in computational complexity with similar or better predictive performance. Our experimental evidence further substantiates that our theoretical work facilitates layer-wise optimal node recovery.

References in corpus (21)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Variational Inference: A Review for Statisticians
• Categorical Reparameterization with Gumbel-Softmax
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Understanding deep learning requires rethinking generalization
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• A correlated topic model of Science
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Group Sparse Regularization for Deep Neural Networks
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Collapse of Deep and Narrow Neural Nets
• Dying ReLU and Initialization: Theory and Numerical Examples
• Convergence rates of posterior distributions for noniid observations
• Learning the Number of Neurons in Deep Networks
• Structured Bayesian Pruning via Log-Normal Multiplicative Noise
• Consistency of Variational Bayes Inference for Estimation and Model Selection in Mixtures
• Posterior Concentration for Sparse Deep Learning
• Convergence Rates of Variational Inference in Sparse Deep Learning
• Efficient Variational Inference for Sparse Deep Learning with Theoretical Guarantee

Cited by in corpus (2)

• Bayesian sparsification for deep neural networks with Bayesian model reduction
• Leveraging Active Subspaces to Capture Epistemic Model Uncertainty in Deep Generative Models for Molecular Design