Model Selection in Bayesian Neural Networks via Horseshoe Priors

Abstract

Bayesian Neural Networks (BNNs) have recently received increasing attention for their ability to provide well-calibrated posterior uncertainties. However, model selection---even choosing the number of nodes---remains an open question. In this work, we apply a horseshoe prior over node pre-activations of a Bayesian neural network, which effectively turns off nodes that do not help explain the data. We demonstrate that our prior prevents the BNN from under-fitting even when the number of nodes required is grossly over-estimated. Moreover, this model selection over the number of nodes doesn't come at the expense of predictive or computational performance; in fact, we learn smaller networks with comparable predictive performance to current approaches.

References in corpus (5)

• Weight Uncertainty in Neural Networks
• Deep Bayesian Active Learning with Image Data
• Group Sparse Regularization for Deep Neural Networks
• Learning Structured Sparsity in Deep Neural Networks
• Automatic Node Selection for Deep Neural Networks using Group Lasso Regularization

Cited by in corpus (5)

• Sparsity information and regularization in the horseshoe and other shrinkage priors
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• Consistent Sparse Deep Learning: Theory and Computation
• Variable Selection with Rigorous Uncertainty Quantification using Deep Bayesian Neural Networks: Posterior Concentration and Bernstein-von Mises Phenomenon
• Combining Model and Parameter Uncertainty in Bayesian Neural Networks