Closed Form Variational Objectives For Bayesian Neural Networks with a Single Hidden Layer
arXiv:1811.00686
Abstract
In this note we consider setups in which variational objectives for Bayesian neural networks can be computed in closed form. In particular we focus on single-layer networks in which the activation function is piecewise polynomial (e.g. ReLU). In this case we show that for a Normal likelihood and structured Normal variational distributions one can compute a variational lower bound in closed form. In addition we compute the predictive mean and variance in closed form. Finally, we also show how to compute approximate lower bounds for other likelihoods (e.g. softmax classification). In experiments we show how the resulting variational objectives can help improve training and provide fast test time predictions.
Bayesian Deep Learning Workshop @ NeurIPS 2018; 11 pages
References in corpus (6)
- Adam: A Method for Stochastic Optimization
- Weight Uncertainty in Neural Networks
- Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
- Variational Auto-encoded Deep Gaussian Processes
- Sampling-Free Variational Inference of Bayesian Neural Networks by Variance Backpropagation
- Probabilistic Meta-Representations Of Neural Networks