Model Selection in Bayesian Neural Networks via Horseshoe Priors

Abstract

Bayesian Neural Networks (BNNs) have recently received increasing attention for their ability to provide well-calibrated posterior uncertainties. However, model selection---even choosing the number of nodes---remains an open question. In this work, we apply a horseshoe prior over node pre-activations of a Bayesian neural network, which effectively turns off nodes that do not help explain the data. We demonstrate that our prior prevents the BNN from under-fitting even when the number of nodes required is grossly over-estimated. Moreover, this model selection over the number of nodes doesn't come at the expense of predictive or computational performance; in fact, we learn smaller networks with comparable predictive performance to current approaches.

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Weight Uncertainty in Neural Networks
• Deep Bayesian Active Learning with Image Data
• Group Sparse Regularization for Deep Neural Networks
• Learning Structured Sparsity in Deep Neural Networks
• Learning the Structure of Deep Sparse Graphical Models
• Auto-Sizing Neural Networks: With Applications to n-gram Language Models
• Automatic Node Selection for Deep Neural Networks using Group Lasso Regularization

Cited by in corpus (16)

• Sparsity information and regularization in the horseshoe and other shrinkage priors
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors
• On the Importance of Strong Baselines in Bayesian Deep Learning
• Action and Perception as Divergence Minimization
• Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Interpretable VAEs for nonlinear group factor analysis
• Consistent Sparse Deep Learning: Theory and Computation
• Variable Selection with Rigorous Uncertainty Quantification using Deep Bayesian Neural Networks: Posterior Concentration and Bernstein-von Mises Phenomenon
• Misclassification bounds for PAC-Bayesian sparse deep learning
• Informative Bayesian Neural Network Priors for Weak Signals
• Combining Model and Parameter Uncertainty in Bayesian Neural Networks
• Sparse encoding for more-interpretable feature-selecting representations in probabilistic matrix factorization
• Generalized Decomposition Priors on R2
• Adaptive Variational Bayesian Inference for Sparse Deep Neural Network